2025-09-07T07:35:07.6535071Z Current runner version: '2.328.0'
2025-09-07T07:35:07.6539980Z Runner name: 'i-0d9447236daff7b96'
2025-09-07T07:35:07.6540771Z Runner group name: 'default'
2025-09-07T07:35:07.6541624Z Machine name: 'ip-10-0-57-198'
2025-09-07T07:35:07.6543782Z ##[group]GITHUB_TOKEN Permissions
2025-09-07T07:35:07.6545678Z Contents: read
2025-09-07T07:35:07.6546300Z Metadata: read
2025-09-07T07:35:07.6546708Z ##[endgroup]
2025-09-07T07:35:07.6548391Z Secret source: Actions
2025-09-07T07:35:07.6548928Z Prepare workflow directory
2025-09-07T07:35:07.6933134Z Prepare all required actions
2025-09-07T07:35:07.6962390Z Getting action download info
2025-09-07T07:35:08.4087844Z Download action repository 'pytorch/test-infra@main' (SHA:548a4bc624d43a01cdf165a63b041f0ae014ddbd)
2025-09-07T07:35:09.4641930Z Download action repository 'pytorch/pytorch@main' (SHA:93fb23d6fae7c4e82c4239a1033e522088742634)
2025-09-07T07:35:23.0925962Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065)
2025-09-07T07:35:23.3385046Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722)
2025-09-07T07:35:23.6070316Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-09-07T07:35:23.7396277Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-09-07T07:35:24.0198874Z Getting action download info
2025-09-07T07:35:24.1125261Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955)
2025-09-07T07:35:24.3658374Z Getting action download info
2025-09-07T07:35:24.4720189Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-09-07T07:35:24.6506543Z Getting action download info
2025-09-07T07:35:24.7806574Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2025-09-07T07:35:25.0517349Z Getting action download info
2025-09-07T07:35:25.1716675Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (93fb23d6fae7c4e82c4239a1033e522088742634)
2025-09-07T07:35:25.1719670Z ##[group] Inputs
2025-09-07T07:35:25.1719951Z   build-environment: linux-jammy-py3.9-gcc11-build
2025-09-07T07:35:25.1722461Z   test-matrix: {"include": [{"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 1, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 2, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 3, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 1, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 2, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 3, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 4, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 5, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 1, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 2, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 3, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 4, "num_shards": 4, "runner": "linux.24xlarge.amd"}]}
2025-09-07T07:35:25.1725756Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:35:25.1726455Z   sync-tag: 
2025-09-07T07:35:25.1727122Z   timeout-minutes: 720
2025-09-07T07:35:25.1727296Z   use-gha: 
2025-09-07T07:35:25.1727691Z   dashboard-tag: training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true
2025-09-07T07:35:25.1728124Z   s3-bucket: gha-artifacts
2025-09-07T07:35:25.1728300Z   aws-role-to-assume: 
2025-09-07T07:35:25.1728688Z   disable-monitor: false
2025-09-07T07:35:25.1728891Z   monitor-log-interval: 15
2025-09-07T07:35:25.1729097Z   monitor-data-collect-interval: 4
2025-09-07T07:35:25.1729300Z ##[endgroup]
2025-09-07T07:35:25.1729628Z Complete job name: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:35:25.2811225Z A job started hook has been configured by the self-hosted runner administrator
2025-09-07T07:35:25.2888576Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2025-09-07T07:35:25.2897118Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:35:25.2897578Z ##[endgroup]
2025-09-07T07:35:26.4664054Z Runner Type: linux.24xlarge.amd
2025-09-07T07:35:26.4664399Z Instance Type: m7a.24xlarge
2025-09-07T07:35:26.4664708Z AMI Name: unknown
2025-09-07T07:35:26.4704034Z AMI ID: ami-05ffe3c48a9991133
2025-09-07T07:35:30.7350398Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2025-09-07T07:35:30.7350733Z with:
2025-09-07T07:35:30.7351190Z   github-secret: ***
2025-09-07T07:35:30.7351638Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2025-09-07T07:35:30.7352101Z   activate-with-label: false
2025-09-07T07:35:30.7352290Z   label: with-ssh
2025-09-07T07:35:30.7352460Z   remove-existing-keys: true
2025-09-07T07:35:30.7352631Z   fail-silently: true
2025-09-07T07:35:30.7352789Z env:
2025-09-07T07:35:30.7352935Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:35:30.7353113Z ##[endgroup]
2025-09-07T07:35:30.8440392Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2025-09-07T07:35:30.8441064Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2025-09-07T07:35:30.8609378Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-09-07T07:35:30.8609678Z with:
2025-09-07T07:35:30.8609815Z   no-sudo: true
2025-09-07T07:35:30.8609985Z   submodules: recursive
2025-09-07T07:35:30.8610149Z   fetch-depth: 0
2025-09-07T07:35:30.8610356Z env:
2025-09-07T07:35:30.8610490Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:35:30.8610684Z ##[endgroup]
2025-09-07T07:35:30.8664336Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-09-07T07:35:30.8664945Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-09-07T07:35:30.8676551Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:35:30.8676809Z env:
2025-09-07T07:35:30.8676983Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:35:30.8677197Z ##[endgroup]
2025-09-07T07:35:30.8761805Z ##[group]Run # Use all available CPUs for fetching
2025-09-07T07:35:30.8762120Z # Use all available CPUs for fetching
2025-09-07T07:35:30.8762356Z cd "${GITHUB_WORKSPACE}"
2025-09-07T07:35:30.8762582Z git config --global fetch.parallel 0
2025-09-07T07:35:30.8762832Z git config --global submodule.fetchJobs 0
2025-09-07T07:35:30.8763049Z 
2025-09-07T07:35:30.8763282Z # Clean workspace. The default checkout action should also do this, but
2025-09-07T07:35:30.8763569Z # do it here as well just in case
2025-09-07T07:35:30.8763792Z if [[ -d .git ]]; then
2025-09-07T07:35:30.8764003Z   if [ -z "${NO_SUDO}" ]; then
2025-09-07T07:35:30.8764204Z     sudo git clean -ffdx
2025-09-07T07:35:30.8764382Z   else
2025-09-07T07:35:30.8764766Z     git clean -ffdx
2025-09-07T07:35:30.8764943Z   fi
2025-09-07T07:35:30.8765095Z fi
2025-09-07T07:35:30.8772204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:35:30.8772449Z env:
2025-09-07T07:35:30.8772682Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:35:30.8772880Z   NO_SUDO: true
2025-09-07T07:35:30.8773053Z ##[endgroup]
2025-09-07T07:35:30.8880652Z ##[group]Run actions/checkout@v4
2025-09-07T07:35:30.8880863Z with:
2025-09-07T07:35:30.8881040Z   ref: 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:35:30.8881254Z   fetch-depth: 0
2025-09-07T07:35:30.8881423Z   submodules: recursive
2025-09-07T07:35:30.8881597Z   show-progress: false
2025-09-07T07:35:30.8881774Z   repository: pytorch/pytorch
2025-09-07T07:35:30.8882032Z   token: ***
2025-09-07T07:35:30.8882183Z   ssh-strict: true
2025-09-07T07:35:30.8882343Z   ssh-user: git
2025-09-07T07:35:30.8882508Z   persist-credentials: true
2025-09-07T07:35:30.8882680Z   clean: true
2025-09-07T07:35:30.8882865Z   sparse-checkout-cone-mode: true
2025-09-07T07:35:30.8883061Z   fetch-tags: false
2025-09-07T07:35:30.8883215Z   lfs: false
2025-09-07T07:35:30.8883356Z   set-safe-directory: true
2025-09-07T07:35:30.8883535Z env:
2025-09-07T07:35:30.8883676Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:35:30.8883839Z ##[endgroup]
2025-09-07T07:35:30.9762172Z Syncing repository: pytorch/pytorch
2025-09-07T07:35:30.9763185Z ##[group]Getting Git version info
2025-09-07T07:35:30.9763512Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2025-09-07T07:35:30.9763941Z [command]/usr/bin/git version
2025-09-07T07:35:30.9986542Z git version 2.47.1
2025-09-07T07:35:31.0006753Z ##[endgroup]
2025-09-07T07:35:31.0015840Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/ee3f3f1a-b755-4d9e-b4d6-8a7a394b087b/.gitconfig'
2025-09-07T07:35:31.0032824Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/ee3f3f1a-b755-4d9e-b4d6-8a7a394b087b' before making global git config changes
2025-09-07T07:35:31.0033443Z Adding repository directory to the temporary git global config as a safe directory
2025-09-07T07:35:31.0037258Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-09-07T07:35:31.0095017Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2025-09-07T07:35:31.0097986Z ##[group]Initializing the repository
2025-09-07T07:35:31.0101318Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-09-07T07:35:31.0167272Z hint: Using 'master' as the name for the initial branch. This default branch name
2025-09-07T07:35:31.0167651Z hint: is subject to change. To configure the initial branch name to use in all
2025-09-07T07:35:31.0168481Z hint: of your new repositories, which will suppress this warning, call:
2025-09-07T07:35:31.0168745Z hint:
2025-09-07T07:35:31.0168977Z hint: 	git config --global init.defaultBranch <name>
2025-09-07T07:35:31.0169213Z hint:
2025-09-07T07:35:31.0169426Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2025-09-07T07:35:31.0169780Z hint: 'development'. The just-created branch can be renamed via this command:
2025-09-07T07:35:31.0170044Z hint:
2025-09-07T07:35:31.0170201Z hint: 	git branch -m <name>
2025-09-07T07:35:31.0187419Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2025-09-07T07:35:31.0196891Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2025-09-07T07:35:31.0240049Z ##[endgroup]
2025-09-07T07:35:31.0240411Z ##[group]Disabling automatic garbage collection
2025-09-07T07:35:31.0242827Z [command]/usr/bin/git config --local gc.auto 0
2025-09-07T07:35:31.0269528Z ##[endgroup]
2025-09-07T07:35:31.0269782Z ##[group]Setting up auth
2025-09-07T07:35:31.0274542Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-09-07T07:35:31.0304368Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-09-07T07:35:31.0711245Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-09-07T07:35:31.0737801Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-09-07T07:35:31.1086759Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-09-07T07:35:31.1130887Z ##[endgroup]
2025-09-07T07:35:31.1131276Z ##[group]Fetching the repository
2025-09-07T07:35:31.1136599Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-09-07T07:36:03.6298253Z From https://github.com/pytorch/pytorch
2025-09-07T07:36:03.6298615Z  * [new branch]              160583                      -> origin/160583
2025-09-07T07:36:03.6299099Z  * [new branch]              2.6.0.dev20241004+          -> origin/2.6.0.dev20241004+
2025-09-07T07:36:03.6299439Z  * [new branch]              5addvllmbuild               -> origin/5addvllmbuild
2025-09-07T07:36:03.6299824Z  * [new branch]              AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest
2025-09-07T07:36:03.6300904Z  * [new branch]              HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes
2025-09-07T07:36:03.6301927Z  * [new branch]              ISSUE-154849                -> origin/ISSUE-154849
2025-09-07T07:36:03.6304498Z  * [new branch]              JackCaoG/dynamo_make_fx_non_core_aten_ops -> origin/JackCaoG/dynamo_make_fx_non_core_aten_ops
2025-09-07T07:36:03.6306255Z  * [new branch]              NicoshevSVE128              -> origin/NicoshevSVE128
2025-09-07T07:36:03.6307597Z  * [new branch]              PR-AOTInductorNoneBug       -> origin/PR-AOTInductorNoneBug
2025-09-07T07:36:03.6308863Z  * [new branch]              PR-AOTInductorNoneBugFix    -> origin/PR-AOTInductorNoneBugFix
2025-09-07T07:36:03.6310935Z  * [new branch]              PR-FixConfigsIssue          -> origin/PR-FixConfigsIssue
2025-09-07T07:36:03.6311300Z  * [new branch]              PR-NoneBugFix-viable        -> origin/PR-NoneBugFix-viable
2025-09-07T07:36:03.6312555Z  * [new branch]              PR-ResetToZero              -> origin/PR-ResetToZero
2025-09-07T07:36:03.6313866Z  * [new branch]              Update-Flash-Packaging      -> origin/Update-Flash-Packaging
2025-09-07T07:36:03.6315084Z  * [new branch]              VLA_exp                     -> origin/VLA_exp
2025-09-07T07:36:03.6316578Z  * [new branch]              actually-run-mps-aot-inductor -> origin/actually-run-mps-aot-inductor
2025-09-07T07:36:03.6317988Z  * [new branch]              add-missing-args-normalization -> origin/add-missing-args-normalization
2025-09-07T07:36:03.6319268Z  * [new branch]              add-user-guide-structure    -> origin/add-user-guide-structure
2025-09-07T07:36:03.6320738Z  * [new branch]              add-vllm-nightly-build      -> origin/add-vllm-nightly-build
2025-09-07T07:36:03.6321902Z  * [new branch]              add_compile_benchmarking    -> origin/add_compile_benchmarking
2025-09-07T07:36:03.6323161Z  * [new branch]              addmm-heuristic             -> origin/addmm-heuristic
2025-09-07T07:36:03.6324447Z  * [new branch]              addsimde                    -> origin/addsimde
2025-09-07T07:36:03.6325738Z  * [new branch]              addvllmtest                 -> origin/addvllmtest
2025-09-07T07:36:03.6327680Z  * [new branch]              adi/acl_upgrade             -> origin/adi/acl_upgrade
2025-09-07T07:36:03.6328914Z  * [new branch]              adi/test                    -> origin/adi/test
2025-09-07T07:36:03.6330221Z  * [new branch]              adi/test_bgemm              -> origin/adi/test_bgemm
2025-09-07T07:36:03.6331607Z  * [new branch]              adi/test_fusions            -> origin/adi/test_fusions
2025-09-07T07:36:03.6335555Z  * [new branch]              adi/test_onednn_v3.9        -> origin/adi/test_onednn_v3.9
2025-09-07T07:36:03.6336212Z  * [new branch]              adi/test_presve_change      -> origin/adi/test_presve_change
2025-09-07T07:36:03.6336548Z  * [new branch]              adi/test_timm               -> origin/adi/test_timm
2025-09-07T07:36:03.6337668Z  * [new branch]              adi/testpresve_change       -> origin/adi/testpresve_change
2025-09-07T07:36:03.6339674Z  * [new branch]              aditew01/test/vec_bf16      -> origin/aditew01/test/vec_bf16
2025-09-07T07:36:03.6340972Z  * [new branch]              ah-globalfeedback-hook      -> origin/ah-globalfeedback-hook
2025-09-07T07:36:03.6342253Z  * [new branch]              alt-disable                 -> origin/alt-disable
2025-09-07T07:36:03.6344177Z  * [new branch]              angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files
2025-09-07T07:36:03.6345409Z  * [new branch]              angelayi/aoti_inductor_fx   -> origin/angelayi/aoti_inductor_fx
2025-09-07T07:36:03.6346559Z  * [new branch]              angelayi/benchmark          -> origin/angelayi/benchmark
2025-09-07T07:36:03.6347921Z  * [new branch]              angelayi/benchmark2         -> origin/angelayi/benchmark2
2025-09-07T07:36:03.6349164Z  * [new branch]              angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization
2025-09-07T07:36:03.6350328Z  * [new branch]              angelayi/cpp_loader         -> origin/angelayi/cpp_loader
2025-09-07T07:36:03.6351999Z  * [new branch]              angelayi/custom_op_subgraph -> origin/angelayi/custom_op_subgraph
2025-09-07T07:36:03.6353566Z  * [new branch]              angelayi/customop           -> origin/angelayi/customop
2025-09-07T07:36:03.6355157Z  * [new branch]              angelayi/fake_cache_empty   -> origin/angelayi/fake_cache_empty
2025-09-07T07:36:03.6356468Z  * [new branch]              angelayi/is_symbolic_tracing -> origin/angelayi/is_symbolic_tracing
2025-09-07T07:36:03.6357673Z  * [new branch]              angelayi/item               -> origin/angelayi/item
2025-09-07T07:36:03.6359075Z  * [new branch]              angelayi/no_so_weight       -> origin/angelayi/no_so_weight
2025-09-07T07:36:03.6360171Z  * [new branch]              angelayi/opoverload         -> origin/angelayi/opoverload
2025-09-07T07:36:03.6361445Z  * [new branch]              angelayi/pattern            -> origin/angelayi/pattern
2025-09-07T07:36:03.6362747Z  * [new branch]              angelayi/pytree             -> origin/angelayi/pytree
2025-09-07T07:36:03.6364022Z  * [new branch]              angelayi/scan_layers        -> origin/angelayi/scan_layers
2025-09-07T07:36:03.6365274Z  * [new branch]              angelayi/symint_input       -> origin/angelayi/symint_input
2025-09-07T07:36:03.6366547Z  * [new branch]              angelayi/test_cpp           -> origin/angelayi/test_cpp
2025-09-07T07:36:03.6367786Z  * [new branch]              angelayi/torch_size         -> origin/angelayi/torch_size
2025-09-07T07:36:03.6369066Z  * [new branch]              aoti-cuda-alloc             -> origin/aoti-cuda-alloc
2025-09-07T07:36:03.6370341Z  * [new branch]              aoti_target_windows         -> origin/aoti_target_windows
2025-09-07T07:36:03.6371590Z  * [new branch]              aoti_weight_sharing         -> origin/aoti_weight_sharing
2025-09-07T07:36:03.6373001Z  * [new branch]              atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124
2025-09-07T07:36:03.6374225Z  * [new branch]              atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1
2025-09-07T07:36:03.6375520Z  * [new branch]              atalman-patch-1             -> origin/atalman-patch-1
2025-09-07T07:36:03.6376846Z  * [new branch]              atalman-patch-3             -> origin/atalman-patch-3
2025-09-07T07:36:03.6378198Z  * [new branch]              atalman-patch-4             -> origin/atalman-patch-4
2025-09-07T07:36:03.6379523Z  * [new branch]              atalman-patch-5             -> origin/atalman-patch-5
2025-09-07T07:36:03.6380798Z  * [new branch]              atalman-patch-6             -> origin/atalman-patch-6
2025-09-07T07:36:03.6382084Z  * [new branch]              atalman_inductor_2.3.0      -> origin/atalman_inductor_2.3.0
2025-09-07T07:36:03.6383304Z  * [new branch]              atalman_inductor_2.3.1      -> origin/atalman_inductor_2.3.1
2025-09-07T07:36:03.6384517Z  * [new branch]              atalman_inductor_2.4.0      -> origin/atalman_inductor_2.4.0
2025-09-07T07:36:03.6385944Z  * [new branch]              atalman_inductor_2.4.x      -> origin/atalman_inductor_2.4.x
2025-09-07T07:36:03.6387331Z  * [new branch]              autoupdate-transformers-pin-via-pr -> origin/autoupdate-transformers-pin-via-pr
2025-09-07T07:36:03.6388996Z  * [new branch]              bahuang/dtensor_demo        -> origin/bahuang/dtensor_demo
2025-09-07T07:36:03.6390227Z  * [new branch]              bahuang/test                -> origin/bahuang/test
2025-09-07T07:36:03.6392267Z  * [new branch]              base/1.5                    -> origin/base/1.5
2025-09-07T07:36:03.6393585Z  * [new branch]              batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention
2025-09-07T07:36:03.6394792Z  * [new branch]              bc-lint-config              -> origin/bc-lint-config
2025-09-07T07:36:03.6396047Z  * [new branch]              bc-lint-test-new-config     -> origin/bc-lint-test-new-config
2025-09-07T07:36:03.6397373Z  * [new branch]              benchmark-updates           -> origin/benchmark-updates
2025-09-07T07:36:03.6398690Z  * [new branch]              benchmarker_compat_with_do_bench -> origin/benchmarker_compat_with_do_bench
2025-09-07T07:36:03.6399934Z  * [new branch]              benchmarking-script         -> origin/benchmarking-script
2025-09-07T07:36:03.6401768Z  * [new branch]              bertmaher/pinbump26         -> origin/bertmaher/pinbump26
2025-09-07T07:36:03.6403802Z  * [new branch]              bertrand/cutlass            -> origin/bertrand/cutlass
2025-09-07T07:36:03.6405593Z  * [new branch]              bf/cg-custom-wrapper        -> origin/bf/cg-custom-wrapper
2025-09-07T07:36:03.6406892Z  * [new branch]              bf/cg-or-error              -> origin/bf/cg-or-error
2025-09-07T07:36:03.6407932Z  * [new branch]              bf/cg-remove-check          -> origin/bf/cg-remove-check
2025-09-07T07:36:03.6409162Z  * [new branch]              bf/cg-skip-1-kernel         -> origin/bf/cg-skip-1-kernel
2025-09-07T07:36:03.6410378Z  * [new branch]              bf/cudagraph                -> origin/bf/cudagraph
2025-09-07T07:36:03.6411870Z  * [new branch]              bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation
2025-09-07T07:36:03.6413656Z  * [new branch]              bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark
2025-09-07T07:36:03.6414705Z  * [new branch]              bf/cudagraph-partition      -> origin/bf/cudagraph-partition
2025-09-07T07:36:03.6416077Z  * [new branch]              bf/default-recompile-reason -> origin/bf/default-recompile-reason
2025-09-07T07:36:03.6417312Z  * [new branch]              bf/donated-buffer-bench     -> origin/bf/donated-buffer-bench
2025-09-07T07:36:03.6418662Z  * [new branch]              bf/exp                      -> origin/bf/exp
2025-09-07T07:36:03.6419939Z  * [new branch]              bf/pa-non-divisible         -> origin/bf/pa-non-divisible
2025-09-07T07:36:03.6421330Z  * [new branch]              bf/partition-move-cpu       -> origin/bf/partition-move-cpu
2025-09-07T07:36:03.6422641Z  * [new branch]              bf/partition-turn-on        -> origin/bf/partition-turn-on
2025-09-07T07:36:03.6423848Z  * [new branch]              bf/remove-check-55b0c39d    -> origin/bf/remove-check-55b0c39d
2025-09-07T07:36:03.6424989Z  * [new branch]              bf/rope                     -> origin/bf/rope
2025-09-07T07:36:03.6426294Z  * [new branch]              bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492
2025-09-07T07:36:03.6427548Z  * [new branch]              bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb
2025-09-07T07:36:03.6428703Z  * [new branch]              bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129
2025-09-07T07:36:03.6429859Z  * [new branch]              bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d
2025-09-07T07:36:03.6431029Z  * [new branch]              bisect_perf_hf_T5_5268754e  -> origin/bisect_perf_hf_T5_5268754e
2025-09-07T07:36:03.6432231Z  * [new branch]              bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c
2025-09-07T07:36:03.6433377Z  * [new branch]              bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c
2025-09-07T07:36:03.6434673Z  * [new branch]              bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f
2025-09-07T07:36:03.6435988Z  * [new branch]              bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0
2025-09-07T07:36:03.6437333Z  * [new branch]              bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149
2025-09-07T07:36:03.6438665Z  * [new branch]              bisect_perf_hf_T5_d65f194a  -> origin/bisect_perf_hf_T5_d65f194a
2025-09-07T07:36:03.6439883Z  * [new branch]              bisect_perf_hf_T5_da94ab0b  -> origin/bisect_perf_hf_T5_da94ab0b
2025-09-07T07:36:03.6441089Z  * [new branch]              bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new
2025-09-07T07:36:03.6442289Z  * [new branch]              bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8
2025-09-07T07:36:03.6443479Z  * [new branch]              bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2
2025-09-07T07:36:03.6444736Z  * [new branch]              bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563
2025-09-07T07:36:03.6446628Z  * [new branch]              bowbao/bench_updates_stage  -> origin/bowbao/bench_updates_stage
2025-09-07T07:36:03.6447867Z  * [new branch]              bowbao/dort_rewriter        -> origin/bowbao/dort_rewriter
2025-09-07T07:36:03.6449070Z  * [new branch]              bowbao/wip_prs              -> origin/bowbao/wip_prs
2025-09-07T07:36:03.6450732Z  * [new branch]              brister/break_tensorbox     -> origin/brister/break_tensorbox
2025-09-07T07:36:03.6451899Z  * [new branch]              brister/custom_fx_backend   -> origin/brister/custom_fx_backend
2025-09-07T07:36:03.6453174Z  * [new branch]              brister/fx_custom_triton    -> origin/brister/fx_custom_triton
2025-09-07T07:36:03.6454319Z  * [new branch]              brister/tensor_box_output   -> origin/brister/tensor_box_output
2025-09-07T07:36:03.6455530Z  * [new branch]              brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check
2025-09-07T07:36:03.6456781Z  * [new branch]              c57382a49                   -> origin/c57382a49
2025-09-07T07:36:03.6458062Z  * [new branch]              ca_0431d47eaa               -> origin/ca_0431d47eaa
2025-09-07T07:36:03.6459272Z  * [new branch]              ca_fix_0431d47eaa           -> origin/ca_fix_0431d47eaa
2025-09-07T07:36:03.6461478Z  * [new branch]              camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8
2025-09-07T07:36:03.6463076Z  * [new branch]              camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push
2025-09-07T07:36:03.6464426Z  * [new branch]              cherry-pick-149654-by-pytorch_bot_bot_ -> origin/cherry-pick-149654-by-pytorch_bot_bot_
2025-09-07T07:36:03.6465701Z  * [new branch]              cherry-pick-151939-by-pytorch_bot_bot_ -> origin/cherry-pick-151939-by-pytorch_bot_bot_
2025-09-07T07:36:03.6467000Z  * [new branch]              cherry-pick-154174-by-pytorch_bot_bot_ -> origin/cherry-pick-154174-by-pytorch_bot_bot_
2025-09-07T07:36:03.6468355Z  * [new branch]              cherry-pick-156260-by-pytorch_bot_bot_ -> origin/cherry-pick-156260-by-pytorch_bot_bot_
2025-09-07T07:36:03.6469614Z  * [new branch]              cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_
2025-09-07T07:36:03.6470954Z  * [new branch]              cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_
2025-09-07T07:36:03.6472148Z  * [new branch]              cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_
2025-09-07T07:36:03.6473383Z  * [new branch]              cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_
2025-09-07T07:36:03.6474595Z  * [new branch]              cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_
2025-09-07T07:36:03.6475914Z  * [new branch]              cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_
2025-09-07T07:36:03.6477310Z  * [new branch]              cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_
2025-09-07T07:36:03.6479031Z  * [new branch]              chilli/flex_vllm            -> origin/chilli/flex_vllm
2025-09-07T07:36:03.6480363Z  * [new branch]              cleanup-inductor-benchmark-images -> origin/cleanup-inductor-benchmark-images
2025-09-07T07:36:03.6481525Z  * [new branch]              codex-testing               -> origin/codex-testing
2025-09-07T07:36:03.6483536Z  * [new branch]              codex/add-helper-function-to-sizevars.py -> origin/codex/add-helper-function-to-sizevars.py
2025-09-07T07:36:03.6484697Z  * [new branch]              codex/add-helper-function-to-sizevars.py_2025-09-05 -> origin/codex/add-helper-function-to-sizevars.py_2025-09-05
2025-09-07T07:36:03.6485808Z  * [new branch]              codex/add-metadata-field-for-file-path -> origin/codex/add-metadata-field-for-file-path
2025-09-07T07:36:03.6487537Z  * [new branch]              codex/add-test-for-inductor-local-cache-behavior -> origin/codex/add-test-for-inductor-local-cache-behavior
2025-09-07T07:36:03.6489400Z  * [new branch]              codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph
2025-09-07T07:36:03.6490448Z  * [new branch]              codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch
2025-09-07T07:36:03.6491666Z  * [new branch]              codex/fix-issue-160415-in-pytorch -> origin/codex/fix-issue-160415-in-pytorch
2025-09-07T07:36:03.6492971Z  * [new branch]              codex/fix-noqengine-quantized-engine-support -> origin/codex/fix-noqengine-quantized-engine-support
2025-09-07T07:36:03.6494139Z  * [new branch]              codex/fix-pin_memory-error-handling -> origin/codex/fix-pin_memory-error-handling
2025-09-07T07:36:03.6495344Z  * [new branch]              codex/propose-fix-for-issue-160332 -> origin/codex/propose-fix-for-issue-160332
2025-09-07T07:36:03.6496647Z  * [new branch]              codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run
2025-09-07T07:36:03.6497906Z  * [new branch]              codex/remove-allow-untyped-defs-and-fix-type-errors -> origin/codex/remove-allow-untyped-defs-and-fix-type-errors
2025-09-07T07:36:03.6499164Z  * [new branch]              compile_fsdp2_disable_stream_and_event -> origin/compile_fsdp2_disable_stream_and_event
2025-09-07T07:36:03.6500264Z  * [new branch]              context_test                -> origin/context_test
2025-09-07T07:36:03.6502078Z  * [new branch]              copilot/fix-157446          -> origin/copilot/fix-157446
2025-09-07T07:36:03.6503536Z  * [new branch]              copy_graph                  -> origin/copy_graph
2025-09-07T07:36:03.6505364Z  * [new branch]              cpio/fix_new_ami_tests      -> origin/cpio/fix_new_ami_tests
2025-09-07T07:36:03.6507126Z  * [new branch]              csl/always_produce_xml      -> origin/csl/always_produce_xml
2025-09-07T07:36:03.6508346Z  * [new branch]              csl/build_test_more_procs   -> origin/csl/build_test_more_procs
2025-09-07T07:36:03.6509548Z  * [new branch]              csl/build_test_more_procs2  -> origin/csl/build_test_more_procs2
2025-09-07T07:36:03.6510732Z  * [new branch]              csl/disable_flaky_cpp_test  -> origin/csl/disable_flaky_cpp_test
2025-09-07T07:36:03.6511877Z  * [new branch]              csl/disable_periodic_test   -> origin/csl/disable_periodic_test
2025-09-07T07:36:03.6513271Z  * [new branch]              csl/exclude_rocm_viable_strict -> origin/csl/exclude_rocm_viable_strict
2025-09-07T07:36:03.6514793Z  * [new branch]              csl/katex                   -> origin/csl/katex
2025-09-07T07:36:03.6516083Z  * [new branch]              csl/larger_runner           -> origin/csl/larger_runner
2025-09-07T07:36:03.6517313Z  * [new branch]              csl/lintrunner_stuff        -> origin/csl/lintrunner_stuff
2025-09-07T07:36:03.6518550Z  * [new branch]              csl/mps_sharding            -> origin/csl/mps_sharding
2025-09-07T07:36:03.6519753Z  * [new branch]              csl/multistage_docker       -> origin/csl/multistage_docker
2025-09-07T07:36:03.6520995Z  * [new branch]              csl/name_link_check_job     -> origin/csl/name_link_check_job
2025-09-07T07:36:03.6522394Z  * [new branch]              csl/no_keep_goin_rocm       -> origin/csl/no_keep_goin_rocm
2025-09-07T07:36:03.6523629Z  * [new branch]              csl/not_600_timeout         -> origin/csl/not_600_timeout
2025-09-07T07:36:03.6524821Z  * [new branch]              csl/revert_open             -> origin/csl/revert_open
2025-09-07T07:36:03.6526089Z  * [new branch]              csl/skip_build              -> origin/csl/skip_build
2025-09-07T07:36:03.6527391Z  * [new branch]              csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner
2025-09-07T07:36:03.6528639Z  * [new branch]              csl/win_sccache             -> origin/csl/win_sccache
2025-09-07T07:36:03.6529892Z  * [new branch]              cublasltrelax2              -> origin/cublasltrelax2
2025-09-07T07:36:03.6531139Z  * [new branch]              cublasrelax2                -> origin/cublasrelax2
2025-09-07T07:36:03.6532511Z  * [new branch]              cudnnsdparefactor           -> origin/cudnnsdparefactor
2025-09-07T07:36:03.6533620Z  * [new branch]              custom_lowering_dict        -> origin/custom_lowering_dict
2025-09-07T07:36:03.6535270Z  * [new branch]              czhuge_muon_dev             -> origin/czhuge_muon_dev
2025-09-07T07:36:03.6537141Z  * [new branch]              d4l3k/delete_hook           -> origin/d4l3k/delete_hook
2025-09-07T07:36:03.6538498Z  * [new branch]              dcp_zoc                     -> origin/dcp_zoc
2025-09-07T07:36:03.6539801Z  * [new branch]              debug-guard                 -> origin/debug-guard
2025-09-07T07:36:03.6541064Z  * [new branch]              delete-quant-docs           -> origin/delete-quant-docs
2025-09-07T07:36:03.6545368Z  * [new branch]              dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.2 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.2
2025-09-07T07:36:03.6546783Z  * [new branch]              dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.3 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.3
2025-09-07T07:36:03.6548125Z  * [new branch]              dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.4 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.55.4
2025-09-07T07:36:03.6549456Z  * [new branch]              dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.56.0 -> origin/dependabot/pip/dot-ci/docker/ci_commit_pins/main/transformers-4.56.0
2025-09-07T07:36:03.6550814Z  * [new branch]              dependabot/pip/dot-ci/docker/protobuf-5.29.5 -> origin/dependabot/pip/dot-ci/docker/protobuf-5.29.5
2025-09-07T07:36:03.6553128Z  * [new branch]              dependabot/pip/dot-github/requirements/protobuf-5.29.5 -> origin/dependabot/pip/dot-github/requirements/protobuf-5.29.5
2025-09-07T07:36:03.6554827Z  * [new branch]              desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper
2025-09-07T07:36:03.6556064Z  * [new branch]              desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64
2025-09-07T07:36:03.6558527Z  * [new branch]              dev/joona/MPSNDArrayAdd     -> origin/dev/joona/MPSNDArrayAdd
2025-09-07T07:36:03.6559842Z  * [new branch]              dev/joona/Unranked          -> origin/dev/joona/Unranked
2025-09-07T07:36:03.6561443Z  * [new branch]              dev/joona/cat               -> origin/dev/joona/cat
2025-09-07T07:36:03.6562750Z  * [new branch]              dev/joona/cat_remove_graph  -> origin/dev/joona/cat_remove_graph
2025-09-07T07:36:03.6563956Z  * [new branch]              dev/joona/embeddingbag      -> origin/dev/joona/embeddingbag
2025-09-07T07:36:03.6565351Z  * [new branch]              dev/joona/getTensorsString  -> origin/dev/joona/getTensorsString
2025-09-07T07:36:03.6566975Z  * [new branch]              dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg
2025-09-07T07:36:03.6568689Z  * [new branch]              dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14
2025-09-07T07:36:03.6570452Z  * [new branch]              dev/joona/sdpa              -> origin/dev/joona/sdpa
2025-09-07T07:36:03.6571940Z  * [new branch]              dev/joona/topk_newapi       -> origin/dev/joona/topk_newapi
2025-09-07T07:36:03.6573330Z  * [new branch]              dev/joona/type_inf          -> origin/dev/joona/type_inf
2025-09-07T07:36:03.6574688Z  * [new branch]              dev/joona/upsize3d          -> origin/dev/joona/upsize3d
2025-09-07T07:36:03.6575886Z  * [new branch]              disable                     -> origin/disable
2025-09-07T07:36:03.6577086Z  * [new branch]              e2e-baseline                -> origin/e2e-baseline
2025-09-07T07:36:03.6578378Z  * [new branch]              eigen_for_sparse_addmm_v2   -> origin/eigen_for_sparse_addmm_v2
2025-09-07T07:36:03.6580327Z  * [new branch]              embg/test_inductor_ci_128B  -> origin/embg/test_inductor_ci_128B
2025-09-07T07:36:03.6581607Z  * [new branch]              embg/test_inductor_ci_base  -> origin/embg/test_inductor_ci_base
2025-09-07T07:36:03.6582765Z  * [new branch]              embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control
2025-09-07T07:36:03.6583896Z  * [new branch]              embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B
2025-09-07T07:36:03.6585294Z  * [new branch]              embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B
2025-09-07T07:36:03.6586689Z  * [new branch]              eqy-patch-1                 -> origin/eqy-patch-1
2025-09-07T07:36:03.6587904Z  * [new branch]              eqy-patch-2                 -> origin/eqy-patch-2
2025-09-07T07:36:03.6589083Z  * [new branch]              eqy-patch-3                 -> origin/eqy-patch-3
2025-09-07T07:36:03.6590278Z  * [new branch]              eqy-patch-4                 -> origin/eqy-patch-4
2025-09-07T07:36:03.6591590Z  * [new branch]              example-convert-torch.nn    -> origin/example-convert-torch.nn
2025-09-07T07:36:03.6593541Z  * [new branch]              exclamaforte/add-contiguous-threshold -> origin/exclamaforte/add-contiguous-threshold
2025-09-07T07:36:03.6594683Z  * [new branch]              exclamaforte/amd-ma         -> origin/exclamaforte/amd-ma
2025-09-07T07:36:03.6596105Z  * [new branch]              exclamaforte/bump-transformer-version -> origin/exclamaforte/bump-transformer-version
2025-09-07T07:36:03.6597321Z  * [new branch]              exclamaforte/clear-feedback-savers -> origin/exclamaforte/clear-feedback-savers
2025-09-07T07:36:03.6598505Z  * [new branch]              exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run
2025-09-07T07:36:03.6599996Z  * [new branch]              exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor
2025-09-07T07:36:03.6601719Z  * [new branch]              exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion
2025-09-07T07:36:03.6603215Z  * [new branch]              exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning
2025-09-07T07:36:03.6604693Z  * [new branch]              exclamaforte/fix-exhuastive-autotuning-reland -> origin/exclamaforte/fix-exhuastive-autotuning-reland
2025-09-07T07:36:03.6605932Z  * [new branch]              exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg
2025-09-07T07:36:03.6607189Z  * [new branch]              exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run
2025-09-07T07:36:03.6608468Z  * [new branch]              exclamaforte/fusion-data    -> origin/exclamaforte/fusion-data
2025-09-07T07:36:03.6609722Z  * [new branch]              exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run
2025-09-07T07:36:03.6611007Z  * [new branch]              exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model
2025-09-07T07:36:03.6612193Z  * [new branch]              exclamaforte/gemm-model     -> origin/exclamaforte/gemm-model
2025-09-07T07:36:03.6613612Z  * [new branch]              exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection
2025-09-07T07:36:03.6614724Z  * [new branch]              exclamaforte/gemm-to-amd    -> origin/exclamaforte/gemm-to-amd
2025-09-07T07:36:03.6615995Z  * [new branch]              exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model
2025-09-07T07:36:03.6617336Z  * [new branch]              exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor
2025-09-07T07:36:03.6618551Z  * [new branch]              exclamaforte/max-autotune-ieee -> origin/exclamaforte/max-autotune-ieee
2025-09-07T07:36:03.6619818Z  * [new branch]              exclamaforte/memory-counter -> origin/exclamaforte/memory-counter
2025-09-07T07:36:03.6621038Z  * [new branch]              exclamaforte/profile-diff-algo -> origin/exclamaforte/profile-diff-algo
2025-09-07T07:36:03.6622243Z  * [new branch]              exclamaforte/profiler-combo -> origin/exclamaforte/profiler-combo
2025-09-07T07:36:03.6623693Z  * [new branch]              exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode
2025-09-07T07:36:03.6624867Z  * [new branch]              exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs
2025-09-07T07:36:03.6626076Z  * [new branch]              exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2
2025-09-07T07:36:03.6627857Z  * [new branch]              exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final
2025-09-07T07:36:03.6629217Z  * [new branch]              exec                        -> origin/exec
2025-09-07T07:36:03.6630456Z  * [new branch]              executorch-module-shim      -> origin/executorch-module-shim
2025-09-07T07:36:03.6631797Z  * [new branch]              experimental-mosaic         -> origin/experimental-mosaic
2025-09-07T07:36:03.6633040Z  * [new branch]              export-D58091437            -> origin/export-D58091437
2025-09-07T07:36:03.6634357Z  * [new branch]              export-D61047529            -> origin/export-D61047529
2025-09-07T07:36:03.6635561Z  * [new branch]              export-D70112642            -> origin/export-D70112642
2025-09-07T07:36:03.6636846Z  * [new branch]              export-D71412006            -> origin/export-D71412006
2025-09-07T07:36:03.6638246Z  * [new branch]              export-D73042989            -> origin/export-D73042989
2025-09-07T07:36:03.6639513Z  * [new branch]              export-D75183591            -> origin/export-D75183591
2025-09-07T07:36:03.6640768Z  * [new branch]              export-D75617432            -> origin/export-D75617432
2025-09-07T07:36:03.6641972Z  * [new branch]              export-D75659965            -> origin/export-D75659965
2025-09-07T07:36:03.6643182Z  * [new branch]              export-D76080931            -> origin/export-D76080931
2025-09-07T07:36:03.6644405Z  * [new branch]              export-D76797250            -> origin/export-D76797250
2025-09-07T07:36:03.6645580Z  * [new branch]              export-D76885271            -> origin/export-D76885271
2025-09-07T07:36:03.6646770Z  * [new branch]              export-D76885620            -> origin/export-D76885620
2025-09-07T07:36:03.6648040Z  * [new branch]              export-D76936623            -> origin/export-D76936623
2025-09-07T07:36:03.6649319Z  * [new branch]              export-D76958268            -> origin/export-D76958268
2025-09-07T07:36:03.6650571Z  * [new branch]              export-D78375400            -> origin/export-D78375400
2025-09-07T07:36:03.6651847Z  * [new branch]              export-D78431305            -> origin/export-D78431305
2025-09-07T07:36:03.6653093Z  * [new branch]              export-D78580107            -> origin/export-D78580107
2025-09-07T07:36:03.6654318Z  * [new branch]              export-D78822171            -> origin/export-D78822171
2025-09-07T07:36:03.6655571Z  * [new branch]              export-D78822351            -> origin/export-D78822351
2025-09-07T07:36:03.6656702Z  * [new branch]              export-D78822507            -> origin/export-D78822507
2025-09-07T07:36:03.6657991Z  * [new branch]              export-D78826994            -> origin/export-D78826994
2025-09-07T07:36:03.6659245Z  * [new branch]              export-D78894324            -> origin/export-D78894324
2025-09-07T07:36:03.6660570Z  * [new branch]              export-D78929245            -> origin/export-D78929245
2025-09-07T07:36:03.6661778Z  * [new branch]              export-D78934925            -> origin/export-D78934925
2025-09-07T07:36:03.6663069Z  * [new branch]              export-D78953203            -> origin/export-D78953203
2025-09-07T07:36:03.6664276Z  * [new branch]              export-D78953229            -> origin/export-D78953229
2025-09-07T07:36:03.6665410Z  * [new branch]              export-D78957093            -> origin/export-D78957093
2025-09-07T07:36:03.6666614Z  * [new branch]              export-D78957389            -> origin/export-D78957389
2025-09-07T07:36:03.6667853Z  * [new branch]              export-D78996107            -> origin/export-D78996107
2025-09-07T07:36:03.6669073Z  * [new branch]              export-D79026433            -> origin/export-D79026433
2025-09-07T07:36:03.6670273Z  * [new branch]              export-D79230339            -> origin/export-D79230339
2025-09-07T07:36:03.6671474Z  * [new branch]              export-D79319835            -> origin/export-D79319835
2025-09-07T07:36:03.6672681Z  * [new branch]              export-D79328456            -> origin/export-D79328456
2025-09-07T07:36:03.6673906Z  * [new branch]              export-D79534608            -> origin/export-D79534608
2025-09-07T07:36:03.6675309Z  * [new branch]              export-D79785974            -> origin/export-D79785974
2025-09-07T07:36:03.6676560Z  * [new branch]              export-D80025417            -> origin/export-D80025417
2025-09-07T07:36:03.6677763Z  * [new branch]              export-D80120333            -> origin/export-D80120333
2025-09-07T07:36:03.6679117Z  * [new branch]              export-D80214882            -> origin/export-D80214882
2025-09-07T07:36:03.6680307Z  * [new branch]              export-D80319069            -> origin/export-D80319069
2025-09-07T07:36:03.6681607Z  * [new branch]              export-D80321215            -> origin/export-D80321215
2025-09-07T07:36:03.6682796Z  * [new branch]              export-D80503451            -> origin/export-D80503451
2025-09-07T07:36:03.6683972Z  * [new branch]              export-D80771648            -> origin/export-D80771648
2025-09-07T07:36:03.6685159Z  * [new branch]              export-D80823877            -> origin/export-D80823877
2025-09-07T07:36:03.6686422Z  * [new branch]              export-D80948073            -> origin/export-D80948073
2025-09-07T07:36:03.6687702Z  * [new branch]              export-D80958642            -> origin/export-D80958642
2025-09-07T07:36:03.6688995Z  * [new branch]              export-D80970483            -> origin/export-D80970483
2025-09-07T07:36:03.6690190Z  * [new branch]              export-D81054193            -> origin/export-D81054193
2025-09-07T07:36:03.6691511Z  * [new branch]              export-D81060182            -> origin/export-D81060182
2025-09-07T07:36:03.6693215Z  * [new branch]              export-D81078973            -> origin/export-D81078973
2025-09-07T07:36:03.6694421Z  * [new branch]              export-D81204584            -> origin/export-D81204584
2025-09-07T07:36:03.6695676Z  * [new branch]              export-D81284190            -> origin/export-D81284190
2025-09-07T07:36:03.6696958Z  * [new branch]              export-D81299840            -> origin/export-D81299840
2025-09-07T07:36:03.6698237Z  * [new branch]              export-D81429090            -> origin/export-D81429090
2025-09-07T07:36:03.6699483Z  * [new branch]              export-D81698719            -> origin/export-D81698719
2025-09-07T07:36:03.6700717Z  * [new branch]              export-D81747409            -> origin/export-D81747409
2025-09-07T07:36:03.6702140Z  * [new branch]              exported-model-train-idempotent -> origin/exported-model-train-idempotent
2025-09-07T07:36:03.6704070Z  * [new branch]              ezyang/wip-aot-descriptors  -> origin/ezyang/wip-aot-descriptors
2025-09-07T07:36:03.6705662Z  * [new branch]              fa_u8_brgemm                -> origin/fa_u8_brgemm
2025-09-07T07:36:03.6706967Z  * [new branch]              fastmath_baseline           -> origin/fastmath_baseline
2025-09-07T07:36:03.6708779Z  * [new branch]              fbcode/warm                 -> origin/fbcode/warm
2025-09-07T07:36:03.6710102Z  * [new branch]              fca                         -> origin/fca
2025-09-07T07:36:03.6711272Z  * [new branch]              fca2_ca5984c                -> origin/fca2_ca5984c
2025-09-07T07:36:03.6712515Z  * [new branch]              fca5                        -> origin/fca5
2025-09-07T07:36:03.6714361Z  * [new branch]              feature/function-numa-binding -> origin/feature/function-numa-binding
2025-09-07T07:36:03.6715666Z  * [new branch]              feature/function-numa-binding-take2 -> origin/feature/function-numa-binding-take2
2025-09-07T07:36:03.6716584Z  * [new branch]              feature/numa-nproc-fix      -> origin/feature/numa-nproc-fix
2025-09-07T07:36:03.6717816Z  * [new branch]              feature/numa-signpost-serialize -> origin/feature/numa-signpost-serialize
2025-09-07T07:36:03.6718945Z  * [new branch]              feature/parallel-numa-binding -> origin/feature/parallel-numa-binding
2025-09-07T07:36:03.6720748Z  * [new branch]              fengyuan/external-proj      -> origin/fengyuan/external-proj
2025-09-07T07:36:03.6721986Z  * [new branch]              fengyuan/out-of-tree-xpu-ops-improve-test -> origin/fengyuan/out-of-tree-xpu-ops-improve-test
2025-09-07T07:36:03.6723081Z  * [new branch]              fengyuan/out-of-tree-xpu-ops-remove-dtype -> origin/fengyuan/out-of-tree-xpu-ops-remove-dtype
2025-09-07T07:36:03.6724175Z  * [new branch]              fengyuan/test-xpu           -> origin/fengyuan/test-xpu
2025-09-07T07:36:03.6725782Z  * [new branch]              ffast_math_baseline         -> origin/ffast_math_baseline
2025-09-07T07:36:03.6727142Z  * [new branch]              ffast_math_target           -> origin/ffast_math_target
2025-09-07T07:36:03.6728936Z  * [new branch]              findhao/base_commit         -> origin/findhao/base_commit
2025-09-07T07:36:03.6730105Z  * [new branch]              findhao/base_commit1        -> origin/findhao/base_commit1
2025-09-07T07:36:03.6731375Z  * [new branch]              findhao/multistream2        -> origin/findhao/multistream2
2025-09-07T07:36:03.6732489Z  * [new branch]              findhao/multistream5        -> origin/findhao/multistream5
2025-09-07T07:36:03.6733678Z  * [new branch]              findhao/multistream6        -> origin/findhao/multistream6
2025-09-07T07:36:03.6734855Z  * [new branch]              findhao/operatorbench3      -> origin/findhao/operatorbench3
2025-09-07T07:36:03.6736014Z  * [new branch]              findhao/operatorbench5      -> origin/findhao/operatorbench5
2025-09-07T07:36:03.6737155Z  * [new branch]              findhao/tritonparse         -> origin/findhao/tritonparse
2025-09-07T07:36:03.6738463Z  * [new branch]              fix                         -> origin/fix
2025-09-07T07:36:03.6739790Z  * [new branch]              fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format
2025-09-07T07:36:03.6740973Z  * [new branch]              fix-config-ignore           -> origin/fix-config-ignore
2025-09-07T07:36:03.6757708Z  * [new branch]              fix-dict-guard              -> origin/fix-dict-guard
2025-09-07T07:36:03.6758314Z  * [new branch]              fix-inductor-periodic-0528  -> origin/fix-inductor-periodic-0528
2025-09-07T07:36:03.6758692Z  * [new branch]              fix-mps-benchmark           -> origin/fix-mps-benchmark
2025-09-07T07:36:03.6759072Z  * [new branch]              fix-rlease-feature-template -> origin/fix-rlease-feature-template
2025-09-07T07:36:03.6759516Z  * [new branch]              fix-run-condition-upload-results -> origin/fix-run-condition-upload-results
2025-09-07T07:36:03.6759907Z  * [new branch]              fix-torchbench              -> origin/fix-torchbench
2025-09-07T07:36:03.6760202Z  * [new branch]              fix_153389                  -> origin/fix_153389
2025-09-07T07:36:03.6760510Z  * [new branch]              fix_fsdp_rs_bucket2         -> origin/fix_fsdp_rs_bucket2
2025-09-07T07:36:03.6760853Z  * [new branch]              fix_inductor_peridic_tests  -> origin/fix_inductor_peridic_tests
2025-09-07T07:36:03.6761181Z  * [new branch]              fix_ubn_159469              -> origin/fix_ubn_159469
2025-09-07T07:36:03.6761475Z  * [new branch]              fixes-triage                -> origin/fixes-triage
2025-09-07T07:36:03.6761771Z  * [new branch]              fixflashinfer               -> origin/fixflashinfer
2025-09-07T07:36:03.6762076Z  * [new branch]              flash_decoding_cpu          -> origin/flash_decoding_cpu
2025-09-07T07:36:03.6762494Z  * [new branch]              flex-flash                  -> origin/flex-flash
2025-09-07T07:36:03.6762800Z  * [new branch]              flex-lowering               -> origin/flex-lowering
2025-09-07T07:36:03.6763092Z  * [new branch]              flex-warning                -> origin/flex-warning
2025-09-07T07:36:03.6763432Z  * [new branch]              flex_attention_functorch_grad -> origin/flex_attention_functorch_grad
2025-09-07T07:36:03.6763768Z  * [new branch]              flex_flash                  -> origin/flex_flash
2025-09-07T07:36:03.6764402Z  * [new branch]              flexdecode-gqa-groups       -> origin/flexdecode-gqa-groups
2025-09-07T07:36:03.6766295Z  * [new branch]              fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule
2025-09-07T07:36:03.6767496Z  * [new branch]              fsdp2_trace_rules           -> origin/fsdp2_trace_rules
2025-09-07T07:36:03.6768745Z  * [new branch]              fsdpv2_3d                   -> origin/fsdpv2_3d
2025-09-07T07:36:03.6770174Z  * [new branch]              fsdpv2_3d_m1                -> origin/fsdpv2_3d_m1
2025-09-07T07:36:03.6771452Z  * [new branch]              fx_cpp                      -> origin/fx_cpp
2025-09-07T07:36:03.6773308Z  * [new branch]              fy/fix-win                  -> origin/fy/fix-win
2025-09-07T07:36:03.6776336Z  * [new branch]              gh/AlnisM/1/base            -> origin/gh/AlnisM/1/base
2025-09-07T07:36:03.6777504Z  * [new branch]              gh/AlnisM/1/head            -> origin/gh/AlnisM/1/head
2025-09-07T07:36:03.6779751Z  * [new branch]              gh/CaoE/2/base              -> origin/gh/CaoE/2/base
2025-09-07T07:36:03.6780949Z  * [new branch]              gh/CaoE/2/head              -> origin/gh/CaoE/2/head
2025-09-07T07:36:03.6782170Z  * [new branch]              gh/CaoE/2/orig              -> origin/gh/CaoE/2/orig
2025-09-07T07:36:03.6784442Z  * [new branch]              gh/ColinPeppler/79/base     -> origin/gh/ColinPeppler/79/base
2025-09-07T07:36:03.6785709Z  * [new branch]              gh/ColinPeppler/79/head     -> origin/gh/ColinPeppler/79/head
2025-09-07T07:36:03.6786911Z  * [new branch]              gh/ColinPeppler/79/orig     -> origin/gh/ColinPeppler/79/orig
2025-09-07T07:36:03.6788808Z  * [new branch]              gh/ColinPeppler/80/base     -> origin/gh/ColinPeppler/80/base
2025-09-07T07:36:03.6790134Z  * [new branch]              gh/ColinPeppler/80/head     -> origin/gh/ColinPeppler/80/head
2025-09-07T07:36:03.6791314Z  * [new branch]              gh/ColinPeppler/80/orig     -> origin/gh/ColinPeppler/80/orig
2025-09-07T07:36:03.6793588Z  * [new branch]              gh/EikanWang/67/base        -> origin/gh/EikanWang/67/base
2025-09-07T07:36:03.6794754Z  * [new branch]              gh/EikanWang/67/head        -> origin/gh/EikanWang/67/head
2025-09-07T07:36:03.6796744Z  * [new branch]              gh/EikanWang/80/base        -> origin/gh/EikanWang/80/base
2025-09-07T07:36:03.6797964Z  * [new branch]              gh/EikanWang/80/head        -> origin/gh/EikanWang/80/head
2025-09-07T07:36:03.6799145Z  * [new branch]              gh/EikanWang/80/orig        -> origin/gh/EikanWang/80/orig
2025-09-07T07:36:03.6800860Z  * [new branch]              gh/EikanWang/81/base        -> origin/gh/EikanWang/81/base
2025-09-07T07:36:03.6802068Z  * [new branch]              gh/EikanWang/81/head        -> origin/gh/EikanWang/81/head
2025-09-07T07:36:03.6803435Z  * [new branch]              gh/EikanWang/81/orig        -> origin/gh/EikanWang/81/orig
2025-09-07T07:36:03.6805447Z  * [new branch]              gh/EikanWang/82/base        -> origin/gh/EikanWang/82/base
2025-09-07T07:36:03.6806683Z  * [new branch]              gh/EikanWang/82/head        -> origin/gh/EikanWang/82/head
2025-09-07T07:36:03.6807927Z  * [new branch]              gh/EikanWang/82/orig        -> origin/gh/EikanWang/82/orig
2025-09-07T07:36:03.6810330Z  * [new branch]              gh/Gasoonjia/1/base         -> origin/gh/Gasoonjia/1/base
2025-09-07T07:36:03.6811551Z  * [new branch]              gh/Gasoonjia/1/head         -> origin/gh/Gasoonjia/1/head
2025-09-07T07:36:03.6813995Z  * [new branch]              gh/H-Huang/131/base         -> origin/gh/H-Huang/131/base
2025-09-07T07:36:03.6815023Z  * [new branch]              gh/H-Huang/131/head         -> origin/gh/H-Huang/131/head
2025-09-07T07:36:03.6816224Z  * [new branch]              gh/H-Huang/131/orig         -> origin/gh/H-Huang/131/orig
2025-09-07T07:36:03.6818110Z  * [new branch]              gh/H-Huang/132/base         -> origin/gh/H-Huang/132/base
2025-09-07T07:36:03.6819423Z  * [new branch]              gh/H-Huang/132/head         -> origin/gh/H-Huang/132/head
2025-09-07T07:36:03.6820612Z  * [new branch]              gh/H-Huang/132/orig         -> origin/gh/H-Huang/132/orig
2025-09-07T07:36:03.6822439Z  * [new branch]              gh/H-Huang/180/base         -> origin/gh/H-Huang/180/base
2025-09-07T07:36:03.6823608Z  * [new branch]              gh/H-Huang/180/head         -> origin/gh/H-Huang/180/head
2025-09-07T07:36:03.6824860Z  * [new branch]              gh/H-Huang/180/orig         -> origin/gh/H-Huang/180/orig
2025-09-07T07:36:03.6826542Z  * [new branch]              gh/H-Huang/182/base         -> origin/gh/H-Huang/182/base
2025-09-07T07:36:03.6827694Z  * [new branch]              gh/H-Huang/182/head         -> origin/gh/H-Huang/182/head
2025-09-07T07:36:03.6828929Z  * [new branch]              gh/H-Huang/182/orig         -> origin/gh/H-Huang/182/orig
2025-09-07T07:36:03.6830752Z  * [new branch]              gh/H-Huang/187/base         -> origin/gh/H-Huang/187/base
2025-09-07T07:36:03.6831907Z  * [new branch]              gh/H-Huang/187/head         -> origin/gh/H-Huang/187/head
2025-09-07T07:36:03.6833190Z  * [new branch]              gh/H-Huang/187/orig         -> origin/gh/H-Huang/187/orig
2025-09-07T07:36:03.6835016Z  * [new branch]              gh/H-Huang/202/base         -> origin/gh/H-Huang/202/base
2025-09-07T07:36:03.6836199Z  * [new branch]              gh/H-Huang/202/head         -> origin/gh/H-Huang/202/head
2025-09-07T07:36:03.6837373Z  * [new branch]              gh/H-Huang/202/orig         -> origin/gh/H-Huang/202/orig
2025-09-07T07:36:03.6839079Z  * [new branch]              gh/H-Huang/203/base         -> origin/gh/H-Huang/203/base
2025-09-07T07:36:03.6840299Z  * [new branch]              gh/H-Huang/203/head         -> origin/gh/H-Huang/203/head
2025-09-07T07:36:03.6841559Z  * [new branch]              gh/H-Huang/203/orig         -> origin/gh/H-Huang/203/orig
2025-09-07T07:36:03.6843351Z  * [new branch]              gh/H-Huang/204/base         -> origin/gh/H-Huang/204/base
2025-09-07T07:36:03.6844586Z  * [new branch]              gh/H-Huang/204/head         -> origin/gh/H-Huang/204/head
2025-09-07T07:36:03.6845769Z  * [new branch]              gh/H-Huang/204/orig         -> origin/gh/H-Huang/204/orig
2025-09-07T07:36:03.6847476Z  * [new branch]              gh/H-Huang/205/base         -> origin/gh/H-Huang/205/base
2025-09-07T07:36:03.6848767Z  * [new branch]              gh/H-Huang/205/head         -> origin/gh/H-Huang/205/head
2025-09-07T07:36:03.6849975Z  * [new branch]              gh/H-Huang/205/orig         -> origin/gh/H-Huang/205/orig
2025-09-07T07:36:03.6851668Z  * [new branch]              gh/H-Huang/206/base         -> origin/gh/H-Huang/206/base
2025-09-07T07:36:03.6852894Z  * [new branch]              gh/H-Huang/206/head         -> origin/gh/H-Huang/206/head
2025-09-07T07:36:03.6854080Z  * [new branch]              gh/H-Huang/206/orig         -> origin/gh/H-Huang/206/orig
2025-09-07T07:36:03.6855816Z  * [new branch]              gh/H-Huang/207/base         -> origin/gh/H-Huang/207/base
2025-09-07T07:36:03.6857018Z  * [new branch]              gh/H-Huang/207/head         -> origin/gh/H-Huang/207/head
2025-09-07T07:36:03.6858381Z  * [new branch]              gh/H-Huang/207/orig         -> origin/gh/H-Huang/207/orig
2025-09-07T07:36:03.6860073Z  * [new branch]              gh/H-Huang/208/base         -> origin/gh/H-Huang/208/base
2025-09-07T07:36:03.6861285Z  * [new branch]              gh/H-Huang/208/head         -> origin/gh/H-Huang/208/head
2025-09-07T07:36:03.6862661Z  * [new branch]              gh/H-Huang/208/orig         -> origin/gh/H-Huang/208/orig
2025-09-07T07:36:03.6864288Z  * [new branch]              gh/H-Huang/209/base         -> origin/gh/H-Huang/209/base
2025-09-07T07:36:03.6865466Z  * [new branch]              gh/H-Huang/209/head         -> origin/gh/H-Huang/209/head
2025-09-07T07:36:03.6866644Z  * [new branch]              gh/H-Huang/209/orig         -> origin/gh/H-Huang/209/orig
2025-09-07T07:36:03.6868481Z  * [new branch]              gh/H-Huang/210/base         -> origin/gh/H-Huang/210/base
2025-09-07T07:36:03.6869652Z  * [new branch]              gh/H-Huang/210/head         -> origin/gh/H-Huang/210/head
2025-09-07T07:36:03.6870841Z  * [new branch]              gh/H-Huang/210/orig         -> origin/gh/H-Huang/210/orig
2025-09-07T07:36:03.6872571Z  * [new branch]              gh/H-Huang/211/base         -> origin/gh/H-Huang/211/base
2025-09-07T07:36:03.6873770Z  * [new branch]              gh/H-Huang/211/head         -> origin/gh/H-Huang/211/head
2025-09-07T07:36:03.6874962Z  * [new branch]              gh/H-Huang/211/orig         -> origin/gh/H-Huang/211/orig
2025-09-07T07:36:03.6876656Z  * [new branch]              gh/H-Huang/212/base         -> origin/gh/H-Huang/212/base
2025-09-07T07:36:03.6877831Z  * [new branch]              gh/H-Huang/212/head         -> origin/gh/H-Huang/212/head
2025-09-07T07:36:03.6879034Z  * [new branch]              gh/H-Huang/212/orig         -> origin/gh/H-Huang/212/orig
2025-09-07T07:36:03.6881726Z  * [new branch]              gh/H-Huang/213/base         -> origin/gh/H-Huang/213/base
2025-09-07T07:36:03.6882981Z  * [new branch]              gh/H-Huang/213/head         -> origin/gh/H-Huang/213/head
2025-09-07T07:36:03.6884138Z  * [new branch]              gh/H-Huang/213/orig         -> origin/gh/H-Huang/213/orig
2025-09-07T07:36:03.6885912Z  * [new branch]              gh/H-Huang/214/base         -> origin/gh/H-Huang/214/base
2025-09-07T07:36:03.6887130Z  * [new branch]              gh/H-Huang/214/head         -> origin/gh/H-Huang/214/head
2025-09-07T07:36:03.6888316Z  * [new branch]              gh/H-Huang/214/orig         -> origin/gh/H-Huang/214/orig
2025-09-07T07:36:03.6890460Z  * [new branch]              gh/IvanKobzarev/112/base    -> origin/gh/IvanKobzarev/112/base
2025-09-07T07:36:03.6891711Z  * [new branch]              gh/IvanKobzarev/112/head    -> origin/gh/IvanKobzarev/112/head
2025-09-07T07:36:03.6892922Z  * [new branch]              gh/IvanKobzarev/112/orig    -> origin/gh/IvanKobzarev/112/orig
2025-09-07T07:36:03.6894740Z  * [new branch]              gh/IvanKobzarev/115/base    -> origin/gh/IvanKobzarev/115/base
2025-09-07T07:36:03.6895983Z  * [new branch]              gh/IvanKobzarev/115/head    -> origin/gh/IvanKobzarev/115/head
2025-09-07T07:36:03.6897265Z  * [new branch]              gh/IvanKobzarev/115/orig    -> origin/gh/IvanKobzarev/115/orig
2025-09-07T07:36:03.6899347Z  * [new branch]              gh/IvanKobzarev/116/base    -> origin/gh/IvanKobzarev/116/base
2025-09-07T07:36:03.6900665Z  * [new branch]              gh/IvanKobzarev/116/head    -> origin/gh/IvanKobzarev/116/head
2025-09-07T07:36:03.6901843Z  * [new branch]              gh/IvanKobzarev/116/orig    -> origin/gh/IvanKobzarev/116/orig
2025-09-07T07:36:03.6903908Z  * [new branch]              gh/IvanKobzarev/118/base    -> origin/gh/IvanKobzarev/118/base
2025-09-07T07:36:03.6905261Z  * [new branch]              gh/IvanKobzarev/118/head    -> origin/gh/IvanKobzarev/118/head
2025-09-07T07:36:03.6906489Z  * [new branch]              gh/IvanKobzarev/118/orig    -> origin/gh/IvanKobzarev/118/orig
2025-09-07T07:36:03.6908428Z  * [new branch]              gh/IvanKobzarev/126/base    -> origin/gh/IvanKobzarev/126/base
2025-09-07T07:36:03.6909671Z  * [new branch]              gh/IvanKobzarev/126/head    -> origin/gh/IvanKobzarev/126/head
2025-09-07T07:36:03.6910878Z  * [new branch]              gh/IvanKobzarev/126/orig    -> origin/gh/IvanKobzarev/126/orig
2025-09-07T07:36:03.6912723Z  * [new branch]              gh/IvanKobzarev/127/base    -> origin/gh/IvanKobzarev/127/base
2025-09-07T07:36:03.6914060Z  * [new branch]              gh/IvanKobzarev/127/head    -> origin/gh/IvanKobzarev/127/head
2025-09-07T07:36:03.6915118Z  * [new branch]              gh/IvanKobzarev/127/orig    -> origin/gh/IvanKobzarev/127/orig
2025-09-07T07:36:03.6916946Z  * [new branch]              gh/IvanKobzarev/128/base    -> origin/gh/IvanKobzarev/128/base
2025-09-07T07:36:03.6918180Z  * [new branch]              gh/IvanKobzarev/128/head    -> origin/gh/IvanKobzarev/128/head
2025-09-07T07:36:03.6919338Z  * [new branch]              gh/IvanKobzarev/128/orig    -> origin/gh/IvanKobzarev/128/orig
2025-09-07T07:36:03.6921179Z  * [new branch]              gh/IvanKobzarev/132/base    -> origin/gh/IvanKobzarev/132/base
2025-09-07T07:36:03.6922432Z  * [new branch]              gh/IvanKobzarev/132/head    -> origin/gh/IvanKobzarev/132/head
2025-09-07T07:36:03.6923661Z  * [new branch]              gh/IvanKobzarev/132/orig    -> origin/gh/IvanKobzarev/132/orig
2025-09-07T07:36:03.6925866Z  * [new branch]              gh/IvanKobzarev/133/base    -> origin/gh/IvanKobzarev/133/base
2025-09-07T07:36:03.6927296Z  * [new branch]              gh/IvanKobzarev/133/head    -> origin/gh/IvanKobzarev/133/head
2025-09-07T07:36:03.6928570Z  * [new branch]              gh/IvanKobzarev/133/orig    -> origin/gh/IvanKobzarev/133/orig
2025-09-07T07:36:03.6930269Z  * [new branch]              gh/IvanKobzarev/134/base    -> origin/gh/IvanKobzarev/134/base
2025-09-07T07:36:03.6931462Z  * [new branch]              gh/IvanKobzarev/134/head    -> origin/gh/IvanKobzarev/134/head
2025-09-07T07:36:03.6932617Z  * [new branch]              gh/IvanKobzarev/134/orig    -> origin/gh/IvanKobzarev/134/orig
2025-09-07T07:36:03.6934615Z  * [new branch]              gh/IvanKobzarev/135/base    -> origin/gh/IvanKobzarev/135/base
2025-09-07T07:36:03.6935811Z  * [new branch]              gh/IvanKobzarev/135/head    -> origin/gh/IvanKobzarev/135/head
2025-09-07T07:36:03.6937027Z  * [new branch]              gh/IvanKobzarev/135/orig    -> origin/gh/IvanKobzarev/135/orig
2025-09-07T07:36:03.6938929Z  * [new branch]              gh/IvanKobzarev/136/base    -> origin/gh/IvanKobzarev/136/base
2025-09-07T07:36:03.6940145Z  * [new branch]              gh/IvanKobzarev/136/head    -> origin/gh/IvanKobzarev/136/head
2025-09-07T07:36:03.6941392Z  * [new branch]              gh/IvanKobzarev/136/orig    -> origin/gh/IvanKobzarev/136/orig
2025-09-07T07:36:03.6943183Z  * [new branch]              gh/IvanKobzarev/137/base    -> origin/gh/IvanKobzarev/137/base
2025-09-07T07:36:03.6944376Z  * [new branch]              gh/IvanKobzarev/137/head    -> origin/gh/IvanKobzarev/137/head
2025-09-07T07:36:03.6945574Z  * [new branch]              gh/IvanKobzarev/137/orig    -> origin/gh/IvanKobzarev/137/orig
2025-09-07T07:36:03.6947311Z  * [new branch]              gh/IvanKobzarev/138/base    -> origin/gh/IvanKobzarev/138/base
2025-09-07T07:36:03.6948526Z  * [new branch]              gh/IvanKobzarev/138/head    -> origin/gh/IvanKobzarev/138/head
2025-09-07T07:36:03.6949847Z  * [new branch]              gh/IvanKobzarev/138/orig    -> origin/gh/IvanKobzarev/138/orig
2025-09-07T07:36:03.6951611Z  * [new branch]              gh/IvanKobzarev/139/base    -> origin/gh/IvanKobzarev/139/base
2025-09-07T07:36:03.6952888Z  * [new branch]              gh/IvanKobzarev/139/head    -> origin/gh/IvanKobzarev/139/head
2025-09-07T07:36:03.6954089Z  * [new branch]              gh/IvanKobzarev/139/orig    -> origin/gh/IvanKobzarev/139/orig
2025-09-07T07:36:03.6955988Z  * [new branch]              gh/IvanKobzarev/140/base    -> origin/gh/IvanKobzarev/140/base
2025-09-07T07:36:03.6957137Z  * [new branch]              gh/IvanKobzarev/140/head    -> origin/gh/IvanKobzarev/140/head
2025-09-07T07:36:03.6958361Z  * [new branch]              gh/IvanKobzarev/140/orig    -> origin/gh/IvanKobzarev/140/orig
2025-09-07T07:36:03.6960671Z  * [new branch]              gh/IvanKobzarev/141/base    -> origin/gh/IvanKobzarev/141/base
2025-09-07T07:36:03.6962105Z  * [new branch]              gh/IvanKobzarev/141/head    -> origin/gh/IvanKobzarev/141/head
2025-09-07T07:36:03.6964418Z  * [new branch]              gh/IvanKobzarev/141/orig    -> origin/gh/IvanKobzarev/141/orig
2025-09-07T07:36:03.6966197Z  * [new branch]              gh/IvanKobzarev/142/base    -> origin/gh/IvanKobzarev/142/base
2025-09-07T07:36:03.6966540Z  * [new branch]              gh/IvanKobzarev/142/head    -> origin/gh/IvanKobzarev/142/head
2025-09-07T07:36:03.6967704Z  * [new branch]              gh/IvanKobzarev/142/orig    -> origin/gh/IvanKobzarev/142/orig
2025-09-07T07:36:03.6969551Z  * [new branch]              gh/IvanKobzarev/143/base    -> origin/gh/IvanKobzarev/143/base
2025-09-07T07:36:03.6970844Z  * [new branch]              gh/IvanKobzarev/143/head    -> origin/gh/IvanKobzarev/143/head
2025-09-07T07:36:03.6972053Z  * [new branch]              gh/IvanKobzarev/143/orig    -> origin/gh/IvanKobzarev/143/orig
2025-09-07T07:36:03.6973952Z  * [new branch]              gh/IvanKobzarev/144/base    -> origin/gh/IvanKobzarev/144/base
2025-09-07T07:36:03.6975147Z  * [new branch]              gh/IvanKobzarev/144/head    -> origin/gh/IvanKobzarev/144/head
2025-09-07T07:36:03.6976320Z  * [new branch]              gh/IvanKobzarev/144/orig    -> origin/gh/IvanKobzarev/144/orig
2025-09-07T07:36:03.6978327Z  * [new branch]              gh/IvanKobzarev/145/base    -> origin/gh/IvanKobzarev/145/base
2025-09-07T07:36:03.6979781Z  * [new branch]              gh/IvanKobzarev/145/head    -> origin/gh/IvanKobzarev/145/head
2025-09-07T07:36:03.6981046Z  * [new branch]              gh/IvanKobzarev/145/orig    -> origin/gh/IvanKobzarev/145/orig
2025-09-07T07:36:03.6982834Z  * [new branch]              gh/IvanKobzarev/146/base    -> origin/gh/IvanKobzarev/146/base
2025-09-07T07:36:03.6984049Z  * [new branch]              gh/IvanKobzarev/146/head    -> origin/gh/IvanKobzarev/146/head
2025-09-07T07:36:03.6985255Z  * [new branch]              gh/IvanKobzarev/146/orig    -> origin/gh/IvanKobzarev/146/orig
2025-09-07T07:36:03.6987401Z  * [new branch]              gh/NikhilAPatel/1/base      -> origin/gh/NikhilAPatel/1/base
2025-09-07T07:36:03.6988694Z  * [new branch]              gh/NikhilAPatel/1/head      -> origin/gh/NikhilAPatel/1/head
2025-09-07T07:36:03.6990343Z  * [new branch]              gh/NikhilAPatel/2/base      -> origin/gh/NikhilAPatel/2/base
2025-09-07T07:36:03.6991511Z  * [new branch]              gh/NikhilAPatel/2/head      -> origin/gh/NikhilAPatel/2/head
2025-09-07T07:36:03.6993418Z  * [new branch]              gh/NikhilAPatel/4/base      -> origin/gh/NikhilAPatel/4/base
2025-09-07T07:36:03.6994759Z  * [new branch]              gh/NikhilAPatel/4/head      -> origin/gh/NikhilAPatel/4/head
2025-09-07T07:36:03.6996793Z  * [new branch]              gh/PaliC/1/base             -> origin/gh/PaliC/1/base
2025-09-07T07:36:03.6998028Z  * [new branch]              gh/PaliC/1/head             -> origin/gh/PaliC/1/head
2025-09-07T07:36:03.6999236Z  * [new branch]              gh/PaliC/1/orig             -> origin/gh/PaliC/1/orig
2025-09-07T07:36:03.7001036Z  * [new branch]              gh/PaliC/17/base            -> origin/gh/PaliC/17/base
2025-09-07T07:36:03.7002211Z  * [new branch]              gh/PaliC/17/head            -> origin/gh/PaliC/17/head
2025-09-07T07:36:03.7003702Z  * [new branch]              gh/PaliC/17/orig            -> origin/gh/PaliC/17/orig
2025-09-07T07:36:03.7005388Z  * [new branch]              gh/PaliC/18/base            -> origin/gh/PaliC/18/base
2025-09-07T07:36:03.7006585Z  * [new branch]              gh/PaliC/18/head            -> origin/gh/PaliC/18/head
2025-09-07T07:36:03.7007769Z  * [new branch]              gh/PaliC/18/orig            -> origin/gh/PaliC/18/orig
2025-09-07T07:36:03.7009455Z  * [new branch]              gh/PaliC/2/base             -> origin/gh/PaliC/2/base
2025-09-07T07:36:03.7010695Z  * [new branch]              gh/PaliC/2/head             -> origin/gh/PaliC/2/head
2025-09-07T07:36:03.7011926Z  * [new branch]              gh/PaliC/2/orig             -> origin/gh/PaliC/2/orig
2025-09-07T07:36:03.7013718Z  * [new branch]              gh/PaliC/20/base            -> origin/gh/PaliC/20/base
2025-09-07T07:36:03.7015051Z  * [new branch]              gh/PaliC/20/head            -> origin/gh/PaliC/20/head
2025-09-07T07:36:03.7016240Z  * [new branch]              gh/PaliC/20/orig            -> origin/gh/PaliC/20/orig
2025-09-07T07:36:03.7018024Z  * [new branch]              gh/PaliC/21/base            -> origin/gh/PaliC/21/base
2025-09-07T07:36:03.7019238Z  * [new branch]              gh/PaliC/21/head            -> origin/gh/PaliC/21/head
2025-09-07T07:36:03.7020454Z  * [new branch]              gh/PaliC/21/orig            -> origin/gh/PaliC/21/orig
2025-09-07T07:36:03.7022142Z  * [new branch]              gh/PaliC/22/base            -> origin/gh/PaliC/22/base
2025-09-07T07:36:03.7023355Z  * [new branch]              gh/PaliC/22/head            -> origin/gh/PaliC/22/head
2025-09-07T07:36:03.7024602Z  * [new branch]              gh/PaliC/22/orig            -> origin/gh/PaliC/22/orig
2025-09-07T07:36:03.7026212Z  * [new branch]              gh/PaliC/23/base            -> origin/gh/PaliC/23/base
2025-09-07T07:36:03.7027483Z  * [new branch]              gh/PaliC/23/head            -> origin/gh/PaliC/23/head
2025-09-07T07:36:03.7028699Z  * [new branch]              gh/PaliC/23/orig            -> origin/gh/PaliC/23/orig
2025-09-07T07:36:03.7030446Z  * [new branch]              gh/PaliC/24/base            -> origin/gh/PaliC/24/base
2025-09-07T07:36:03.7031614Z  * [new branch]              gh/PaliC/24/head            -> origin/gh/PaliC/24/head
2025-09-07T07:36:03.7032801Z  * [new branch]              gh/PaliC/24/orig            -> origin/gh/PaliC/24/orig
2025-09-07T07:36:03.7034917Z  * [new branch]              gh/PaulZhang12/17/base      -> origin/gh/PaulZhang12/17/base
2025-09-07T07:36:03.7036115Z  * [new branch]              gh/PaulZhang12/17/head      -> origin/gh/PaulZhang12/17/head
2025-09-07T07:36:03.7038001Z  * [new branch]              gh/PaulZhang12/20/base      -> origin/gh/PaulZhang12/20/base
2025-09-07T07:36:03.7039206Z  * [new branch]              gh/PaulZhang12/20/head      -> origin/gh/PaulZhang12/20/head
2025-09-07T07:36:03.7040450Z  * [new branch]              gh/PaulZhang12/20/orig      -> origin/gh/PaulZhang12/20/orig
2025-09-07T07:36:03.7042174Z  * [new branch]              gh/PaulZhang12/21/base      -> origin/gh/PaulZhang12/21/base
2025-09-07T07:36:03.7043506Z  * [new branch]              gh/PaulZhang12/21/head      -> origin/gh/PaulZhang12/21/head
2025-09-07T07:36:03.7044783Z  * [new branch]              gh/PaulZhang12/21/orig      -> origin/gh/PaulZhang12/21/orig
2025-09-07T07:36:03.7046515Z  * [new branch]              gh/PaulZhang12/22/base      -> origin/gh/PaulZhang12/22/base
2025-09-07T07:36:03.7047714Z  * [new branch]              gh/PaulZhang12/22/head      -> origin/gh/PaulZhang12/22/head
2025-09-07T07:36:03.7048892Z  * [new branch]              gh/PaulZhang12/22/orig      -> origin/gh/PaulZhang12/22/orig
2025-09-07T07:36:03.7050630Z  * [new branch]              gh/PaulZhang12/23/base      -> origin/gh/PaulZhang12/23/base
2025-09-07T07:36:03.7051905Z  * [new branch]              gh/PaulZhang12/23/head      -> origin/gh/PaulZhang12/23/head
2025-09-07T07:36:03.7053122Z  * [new branch]              gh/PaulZhang12/23/orig      -> origin/gh/PaulZhang12/23/orig
2025-09-07T07:36:03.7054702Z  * [new branch]              gh/PaulZhang12/24/base      -> origin/gh/PaulZhang12/24/base
2025-09-07T07:36:03.7055917Z  * [new branch]              gh/PaulZhang12/24/head      -> origin/gh/PaulZhang12/24/head
2025-09-07T07:36:03.7057192Z  * [new branch]              gh/PaulZhang12/24/orig      -> origin/gh/PaulZhang12/24/orig
2025-09-07T07:36:03.7058974Z  * [new branch]              gh/PaulZhang12/25/base      -> origin/gh/PaulZhang12/25/base
2025-09-07T07:36:03.7060246Z  * [new branch]              gh/PaulZhang12/25/head      -> origin/gh/PaulZhang12/25/head
2025-09-07T07:36:03.7061597Z  * [new branch]              gh/PaulZhang12/25/orig      -> origin/gh/PaulZhang12/25/orig
2025-09-07T07:36:03.7063693Z  * [new branch]              gh/SamGinzburg/11/base      -> origin/gh/SamGinzburg/11/base
2025-09-07T07:36:03.7065061Z  * [new branch]              gh/SamGinzburg/11/head      -> origin/gh/SamGinzburg/11/head
2025-09-07T07:36:03.7067903Z  * [new branch]              gh/Sidharth123-cpu/24/base  -> origin/gh/Sidharth123-cpu/24/base
2025-09-07T07:36:03.7069576Z  * [new branch]              gh/Sidharth123-cpu/25/base  -> origin/gh/Sidharth123-cpu/25/base
2025-09-07T07:36:03.7071168Z  * [new branch]              gh/Sidharth123-cpu/26/base  -> origin/gh/Sidharth123-cpu/26/base
2025-09-07T07:36:03.7072969Z  * [new branch]              gh/Sidharth123-cpu/27/base  -> origin/gh/Sidharth123-cpu/27/base
2025-09-07T07:36:03.7075170Z  * [new branch]              gh/StrongerXi/1/base        -> origin/gh/StrongerXi/1/base
2025-09-07T07:36:03.7076378Z  * [new branch]              gh/StrongerXi/1/head        -> origin/gh/StrongerXi/1/head
2025-09-07T07:36:03.7078126Z  * [new branch]              gh/StrongerXi/133/base      -> origin/gh/StrongerXi/133/base
2025-09-07T07:36:03.7079319Z  * [new branch]              gh/StrongerXi/133/head      -> origin/gh/StrongerXi/133/head
2025-09-07T07:36:03.7080546Z  * [new branch]              gh/StrongerXi/133/orig      -> origin/gh/StrongerXi/133/orig
2025-09-07T07:36:03.7082231Z  * [new branch]              gh/StrongerXi/134/base      -> origin/gh/StrongerXi/134/base
2025-09-07T07:36:03.7083414Z  * [new branch]              gh/StrongerXi/134/head      -> origin/gh/StrongerXi/134/head
2025-09-07T07:36:03.7084614Z  * [new branch]              gh/StrongerXi/134/orig      -> origin/gh/StrongerXi/134/orig
2025-09-07T07:36:03.7086328Z  * [new branch]              gh/StrongerXi/136/base      -> origin/gh/StrongerXi/136/base
2025-09-07T07:36:03.7087506Z  * [new branch]              gh/StrongerXi/136/head      -> origin/gh/StrongerXi/136/head
2025-09-07T07:36:03.7088721Z  * [new branch]              gh/StrongerXi/136/orig      -> origin/gh/StrongerXi/136/orig
2025-09-07T07:36:03.7090564Z  * [new branch]              gh/StrongerXi/137/base      -> origin/gh/StrongerXi/137/base
2025-09-07T07:36:03.7091870Z  * [new branch]              gh/StrongerXi/137/head      -> origin/gh/StrongerXi/137/head
2025-09-07T07:36:03.7093055Z  * [new branch]              gh/StrongerXi/137/orig      -> origin/gh/StrongerXi/137/orig
2025-09-07T07:36:03.7094768Z  * [new branch]              gh/StrongerXi/138/base      -> origin/gh/StrongerXi/138/base
2025-09-07T07:36:03.7096001Z  * [new branch]              gh/StrongerXi/138/head      -> origin/gh/StrongerXi/138/head
2025-09-07T07:36:03.7097221Z  * [new branch]              gh/StrongerXi/138/orig      -> origin/gh/StrongerXi/138/orig
2025-09-07T07:36:03.7099027Z  * [new branch]              gh/StrongerXi/139/base      -> origin/gh/StrongerXi/139/base
2025-09-07T07:36:03.7100224Z  * [new branch]              gh/StrongerXi/139/head      -> origin/gh/StrongerXi/139/head
2025-09-07T07:36:03.7101499Z  * [new branch]              gh/StrongerXi/139/orig      -> origin/gh/StrongerXi/139/orig
2025-09-07T07:36:03.7103410Z  * [new branch]              gh/StrongerXi/140/base      -> origin/gh/StrongerXi/140/base
2025-09-07T07:36:03.7104654Z  * [new branch]              gh/StrongerXi/140/head      -> origin/gh/StrongerXi/140/head
2025-09-07T07:36:03.7105984Z  * [new branch]              gh/StrongerXi/140/orig      -> origin/gh/StrongerXi/140/orig
2025-09-07T07:36:03.7107707Z  * [new branch]              gh/StrongerXi/71/base       -> origin/gh/StrongerXi/71/base
2025-09-07T07:36:03.7108961Z  * [new branch]              gh/StrongerXi/71/head       -> origin/gh/StrongerXi/71/head
2025-09-07T07:36:03.7110624Z  * [new branch]              gh/StrongerXi/72/base       -> origin/gh/StrongerXi/72/base
2025-09-07T07:36:03.7111856Z  * [new branch]              gh/StrongerXi/72/head       -> origin/gh/StrongerXi/72/head
2025-09-07T07:36:03.7114082Z  * [new branch]              gh/XilunWu/133/base         -> origin/gh/XilunWu/133/base
2025-09-07T07:36:03.7115215Z  * [new branch]              gh/XilunWu/133/head         -> origin/gh/XilunWu/133/head
2025-09-07T07:36:03.7116570Z  * [new branch]              gh/XilunWu/133/orig         -> origin/gh/XilunWu/133/orig
2025-09-07T07:36:03.7118211Z  * [new branch]              gh/XilunWu/139/base         -> origin/gh/XilunWu/139/base
2025-09-07T07:36:03.7119400Z  * [new branch]              gh/XilunWu/139/head         -> origin/gh/XilunWu/139/head
2025-09-07T07:36:03.7120514Z  * [new branch]              gh/XilunWu/139/orig         -> origin/gh/XilunWu/139/orig
2025-09-07T07:36:03.7122356Z  * [new branch]              gh/XilunWu/143/base         -> origin/gh/XilunWu/143/base
2025-09-07T07:36:03.7123614Z  * [new branch]              gh/XilunWu/143/head         -> origin/gh/XilunWu/143/head
2025-09-07T07:36:03.7124900Z  * [new branch]              gh/XilunWu/143/orig         -> origin/gh/XilunWu/143/orig
2025-09-07T07:36:03.7126827Z  * [new branch]              gh/XilunWu/144/base         -> origin/gh/XilunWu/144/base
2025-09-07T07:36:03.7128011Z  * [new branch]              gh/XilunWu/144/head         -> origin/gh/XilunWu/144/head
2025-09-07T07:36:03.7129234Z  * [new branch]              gh/XilunWu/144/orig         -> origin/gh/XilunWu/144/orig
2025-09-07T07:36:03.7130976Z  * [new branch]              gh/XilunWu/145/base         -> origin/gh/XilunWu/145/base
2025-09-07T07:36:03.7132128Z  * [new branch]              gh/XilunWu/145/head         -> origin/gh/XilunWu/145/head
2025-09-07T07:36:03.7133315Z  * [new branch]              gh/XilunWu/145/orig         -> origin/gh/XilunWu/145/orig
2025-09-07T07:36:03.7134946Z  * [new branch]              gh/XilunWu/146/base         -> origin/gh/XilunWu/146/base
2025-09-07T07:36:03.7136213Z  * [new branch]              gh/XilunWu/146/head         -> origin/gh/XilunWu/146/head
2025-09-07T07:36:03.7137429Z  * [new branch]              gh/XilunWu/146/orig         -> origin/gh/XilunWu/146/orig
2025-09-07T07:36:03.7139238Z  * [new branch]              gh/XilunWu/147/base         -> origin/gh/XilunWu/147/base
2025-09-07T07:36:03.7140400Z  * [new branch]              gh/XilunWu/147/head         -> origin/gh/XilunWu/147/head
2025-09-07T07:36:03.7141652Z  * [new branch]              gh/XilunWu/147/orig         -> origin/gh/XilunWu/147/orig
2025-09-07T07:36:03.7143235Z  * [new branch]              gh/XilunWu/148/base         -> origin/gh/XilunWu/148/base
2025-09-07T07:36:03.7144465Z  * [new branch]              gh/XilunWu/148/head         -> origin/gh/XilunWu/148/head
2025-09-07T07:36:03.7145627Z  * [new branch]              gh/XilunWu/148/orig         -> origin/gh/XilunWu/148/orig
2025-09-07T07:36:03.7147223Z  * [new branch]              gh/XilunWu/149/base         -> origin/gh/XilunWu/149/base
2025-09-07T07:36:03.7148407Z  * [new branch]              gh/XilunWu/149/head         -> origin/gh/XilunWu/149/head
2025-09-07T07:36:03.7149643Z  * [new branch]              gh/XilunWu/149/orig         -> origin/gh/XilunWu/149/orig
2025-09-07T07:36:03.7151226Z  * [new branch]              gh/XilunWu/150/base         -> origin/gh/XilunWu/150/base
2025-09-07T07:36:03.7152436Z  * [new branch]              gh/XilunWu/150/head         -> origin/gh/XilunWu/150/head
2025-09-07T07:36:03.7153665Z  * [new branch]              gh/XilunWu/150/orig         -> origin/gh/XilunWu/150/orig
2025-09-07T07:36:03.7155361Z  * [new branch]              gh/XilunWu/151/base         -> origin/gh/XilunWu/151/base
2025-09-07T07:36:03.7156570Z  * [new branch]              gh/XilunWu/151/head         -> origin/gh/XilunWu/151/head
2025-09-07T07:36:03.7157897Z  * [new branch]              gh/XilunWu/151/orig         -> origin/gh/XilunWu/151/orig
2025-09-07T07:36:03.7159515Z  * [new branch]              gh/XilunWu/152/base         -> origin/gh/XilunWu/152/base
2025-09-07T07:36:03.7160651Z  * [new branch]              gh/XilunWu/152/head         -> origin/gh/XilunWu/152/head
2025-09-07T07:36:03.7161875Z  * [new branch]              gh/XilunWu/152/orig         -> origin/gh/XilunWu/152/orig
2025-09-07T07:36:03.7163777Z  * [new branch]              gh/XilunWu/153/base         -> origin/gh/XilunWu/153/base
2025-09-07T07:36:03.7165150Z  * [new branch]              gh/XilunWu/153/head         -> origin/gh/XilunWu/153/head
2025-09-07T07:36:03.7166339Z  * [new branch]              gh/XilunWu/153/orig         -> origin/gh/XilunWu/153/orig
2025-09-07T07:36:03.7168119Z  * [new branch]              gh/XilunWu/160/base         -> origin/gh/XilunWu/160/base
2025-09-07T07:36:03.7169340Z  * [new branch]              gh/XilunWu/160/head         -> origin/gh/XilunWu/160/head
2025-09-07T07:36:03.7170730Z  * [new branch]              gh/XilunWu/160/orig         -> origin/gh/XilunWu/160/orig
2025-09-07T07:36:03.7172705Z  * [new branch]              gh/XilunWu/161/base         -> origin/gh/XilunWu/161/base
2025-09-07T07:36:03.7173897Z  * [new branch]              gh/XilunWu/161/head         -> origin/gh/XilunWu/161/head
2025-09-07T07:36:03.7175173Z  * [new branch]              gh/XilunWu/161/orig         -> origin/gh/XilunWu/161/orig
2025-09-07T07:36:03.7177054Z  * [new branch]              gh/XilunWu/163/base         -> origin/gh/XilunWu/163/base
2025-09-07T07:36:03.7178298Z  * [new branch]              gh/XilunWu/163/head         -> origin/gh/XilunWu/163/head
2025-09-07T07:36:03.7179612Z  * [new branch]              gh/XilunWu/163/orig         -> origin/gh/XilunWu/163/orig
2025-09-07T07:36:03.7182035Z  * [new branch]              gh/XilunWu/164/base         -> origin/gh/XilunWu/164/base
2025-09-07T07:36:03.7183316Z  * [new branch]              gh/XilunWu/164/head         -> origin/gh/XilunWu/164/head
2025-09-07T07:36:03.7184538Z  * [new branch]              gh/XilunWu/164/orig         -> origin/gh/XilunWu/164/orig
2025-09-07T07:36:03.7186447Z  * [new branch]              gh/XilunWu/165/base         -> origin/gh/XilunWu/165/base
2025-09-07T07:36:03.7187723Z  * [new branch]              gh/XilunWu/165/head         -> origin/gh/XilunWu/165/head
2025-09-07T07:36:03.7188953Z  * [new branch]              gh/XilunWu/165/orig         -> origin/gh/XilunWu/165/orig
2025-09-07T07:36:03.7190850Z  * [new branch]              gh/XilunWu/166/base         -> origin/gh/XilunWu/166/base
2025-09-07T07:36:03.7192109Z  * [new branch]              gh/XilunWu/166/head         -> origin/gh/XilunWu/166/head
2025-09-07T07:36:03.7193385Z  * [new branch]              gh/XilunWu/166/orig         -> origin/gh/XilunWu/166/orig
2025-09-07T07:36:03.7195179Z  * [new branch]              gh/XilunWu/167/base         -> origin/gh/XilunWu/167/base
2025-09-07T07:36:03.7196420Z  * [new branch]              gh/XilunWu/167/head         -> origin/gh/XilunWu/167/head
2025-09-07T07:36:03.7197662Z  * [new branch]              gh/XilunWu/167/orig         -> origin/gh/XilunWu/167/orig
2025-09-07T07:36:03.7199774Z  * [new branch]              gh/XilunWu/168/base         -> origin/gh/XilunWu/168/base
2025-09-07T07:36:03.7200970Z  * [new branch]              gh/XilunWu/168/head         -> origin/gh/XilunWu/168/head
2025-09-07T07:36:03.7202210Z  * [new branch]              gh/XilunWu/168/orig         -> origin/gh/XilunWu/168/orig
2025-09-07T07:36:03.7204229Z  * [new branch]              gh/XilunWu/169/base         -> origin/gh/XilunWu/169/base
2025-09-07T07:36:03.7205426Z  * [new branch]              gh/XilunWu/169/head         -> origin/gh/XilunWu/169/head
2025-09-07T07:36:03.7206594Z  * [new branch]              gh/XilunWu/169/orig         -> origin/gh/XilunWu/169/orig
2025-09-07T07:36:03.7208271Z  * [new branch]              gh/XilunWu/170/base         -> origin/gh/XilunWu/170/base
2025-09-07T07:36:03.7209476Z  * [new branch]              gh/XilunWu/170/head         -> origin/gh/XilunWu/170/head
2025-09-07T07:36:03.7210745Z  * [new branch]              gh/XilunWu/170/orig         -> origin/gh/XilunWu/170/orig
2025-09-07T07:36:03.7212978Z  * [new branch]              gh/XuehaiPan/14/base        -> origin/gh/XuehaiPan/14/base
2025-09-07T07:36:03.7214241Z  * [new branch]              gh/XuehaiPan/14/head        -> origin/gh/XuehaiPan/14/head
2025-09-07T07:36:03.7215361Z  * [new branch]              gh/XuehaiPan/14/orig        -> origin/gh/XuehaiPan/14/orig
2025-09-07T07:36:03.7217200Z  * [new branch]              gh/XuehaiPan/179/base       -> origin/gh/XuehaiPan/179/base
2025-09-07T07:36:03.7218584Z  * [new branch]              gh/XuehaiPan/179/head       -> origin/gh/XuehaiPan/179/head
2025-09-07T07:36:03.7219774Z  * [new branch]              gh/XuehaiPan/179/orig       -> origin/gh/XuehaiPan/179/orig
2025-09-07T07:36:03.7221671Z  * [new branch]              gh/XuehaiPan/189/base       -> origin/gh/XuehaiPan/189/base
2025-09-07T07:36:03.7222835Z  * [new branch]              gh/XuehaiPan/189/head       -> origin/gh/XuehaiPan/189/head
2025-09-07T07:36:03.7224025Z  * [new branch]              gh/XuehaiPan/189/orig       -> origin/gh/XuehaiPan/189/orig
2025-09-07T07:36:03.7225801Z  * [new branch]              gh/XuehaiPan/232/base       -> origin/gh/XuehaiPan/232/base
2025-09-07T07:36:03.7227118Z  * [new branch]              gh/XuehaiPan/232/head       -> origin/gh/XuehaiPan/232/head
2025-09-07T07:36:03.7228310Z  * [new branch]              gh/XuehaiPan/232/orig       -> origin/gh/XuehaiPan/232/orig
2025-09-07T07:36:03.7230041Z  * [new branch]              gh/XuehaiPan/249/base       -> origin/gh/XuehaiPan/249/base
2025-09-07T07:36:03.7231287Z  * [new branch]              gh/XuehaiPan/249/head       -> origin/gh/XuehaiPan/249/head
2025-09-07T07:36:03.7232439Z  * [new branch]              gh/XuehaiPan/249/orig       -> origin/gh/XuehaiPan/249/orig
2025-09-07T07:36:03.7234101Z  * [new branch]              gh/XuehaiPan/253/base       -> origin/gh/XuehaiPan/253/base
2025-09-07T07:36:03.7235707Z  * [new branch]              gh/XuehaiPan/253/head       -> origin/gh/XuehaiPan/253/head
2025-09-07T07:36:03.7236671Z  * [new branch]              gh/XuehaiPan/253/orig       -> origin/gh/XuehaiPan/253/orig
2025-09-07T07:36:03.7238295Z  * [new branch]              gh/XuehaiPan/254/base       -> origin/gh/XuehaiPan/254/base
2025-09-07T07:36:03.7239519Z  * [new branch]              gh/XuehaiPan/254/head       -> origin/gh/XuehaiPan/254/head
2025-09-07T07:36:03.7240801Z  * [new branch]              gh/XuehaiPan/254/orig       -> origin/gh/XuehaiPan/254/orig
2025-09-07T07:36:03.7242503Z  * [new branch]              gh/XuehaiPan/255/base       -> origin/gh/XuehaiPan/255/base
2025-09-07T07:36:03.7243744Z  * [new branch]              gh/XuehaiPan/255/head       -> origin/gh/XuehaiPan/255/head
2025-09-07T07:36:03.7244862Z  * [new branch]              gh/XuehaiPan/255/orig       -> origin/gh/XuehaiPan/255/orig
2025-09-07T07:36:03.7246592Z  * [new branch]              gh/XuehaiPan/257/base       -> origin/gh/XuehaiPan/257/base
2025-09-07T07:36:03.7247785Z  * [new branch]              gh/XuehaiPan/257/head       -> origin/gh/XuehaiPan/257/head
2025-09-07T07:36:03.7248992Z  * [new branch]              gh/XuehaiPan/257/orig       -> origin/gh/XuehaiPan/257/orig
2025-09-07T07:36:03.7250702Z  * [new branch]              gh/XuehaiPan/271/base       -> origin/gh/XuehaiPan/271/base
2025-09-07T07:36:03.7251866Z  * [new branch]              gh/XuehaiPan/271/head       -> origin/gh/XuehaiPan/271/head
2025-09-07T07:36:03.7253064Z  * [new branch]              gh/XuehaiPan/271/orig       -> origin/gh/XuehaiPan/271/orig
2025-09-07T07:36:03.7254751Z  * [new branch]              gh/XuehaiPan/290/base       -> origin/gh/XuehaiPan/290/base
2025-09-07T07:36:03.7256069Z  * [new branch]              gh/XuehaiPan/290/head       -> origin/gh/XuehaiPan/290/head
2025-09-07T07:36:03.7257183Z  * [new branch]              gh/XuehaiPan/290/orig       -> origin/gh/XuehaiPan/290/orig
2025-09-07T07:36:03.7259014Z  * [new branch]              gh/XuehaiPan/343/base       -> origin/gh/XuehaiPan/343/base
2025-09-07T07:36:03.7266943Z  * [new branch]              gh/XuehaiPan/343/head       -> origin/gh/XuehaiPan/343/head
2025-09-07T07:36:03.7267286Z  * [new branch]              gh/XuehaiPan/343/orig       -> origin/gh/XuehaiPan/343/orig
2025-09-07T07:36:03.7267601Z  * [new branch]              gh/XuehaiPan/347/base       -> origin/gh/XuehaiPan/347/base
2025-09-07T07:36:03.7267927Z  * [new branch]              gh/XuehaiPan/347/head       -> origin/gh/XuehaiPan/347/head
2025-09-07T07:36:03.7268320Z  * [new branch]              gh/XuehaiPan/347/orig       -> origin/gh/XuehaiPan/347/orig
2025-09-07T07:36:03.7268650Z  * [new branch]              gh/XuehaiPan/348/base       -> origin/gh/XuehaiPan/348/base
2025-09-07T07:36:03.7268973Z  * [new branch]              gh/XuehaiPan/348/head       -> origin/gh/XuehaiPan/348/head
2025-09-07T07:36:03.7270069Z  * [new branch]              gh/XuehaiPan/348/orig       -> origin/gh/XuehaiPan/348/orig
2025-09-07T07:36:03.7271757Z  * [new branch]              gh/XuehaiPan/350/base       -> origin/gh/XuehaiPan/350/base
2025-09-07T07:36:03.7272958Z  * [new branch]              gh/XuehaiPan/350/head       -> origin/gh/XuehaiPan/350/head
2025-09-07T07:36:03.7274230Z  * [new branch]              gh/XuehaiPan/350/orig       -> origin/gh/XuehaiPan/350/orig
2025-09-07T07:36:03.7276082Z  * [new branch]              gh/XuehaiPan/356/base       -> origin/gh/XuehaiPan/356/base
2025-09-07T07:36:03.7277507Z  * [new branch]              gh/XuehaiPan/356/head       -> origin/gh/XuehaiPan/356/head
2025-09-07T07:36:03.7278751Z  * [new branch]              gh/XuehaiPan/356/orig       -> origin/gh/XuehaiPan/356/orig
2025-09-07T07:36:03.7280420Z  * [new branch]              gh/XuehaiPan/357/base       -> origin/gh/XuehaiPan/357/base
2025-09-07T07:36:03.7281620Z  * [new branch]              gh/XuehaiPan/357/head       -> origin/gh/XuehaiPan/357/head
2025-09-07T07:36:03.7282792Z  * [new branch]              gh/XuehaiPan/357/orig       -> origin/gh/XuehaiPan/357/orig
2025-09-07T07:36:03.7284966Z  * [new branch]              gh/XuehaiPan/358/base       -> origin/gh/XuehaiPan/358/base
2025-09-07T07:36:03.7286192Z  * [new branch]              gh/XuehaiPan/358/head       -> origin/gh/XuehaiPan/358/head
2025-09-07T07:36:03.7287431Z  * [new branch]              gh/XuehaiPan/358/orig       -> origin/gh/XuehaiPan/358/orig
2025-09-07T07:36:03.7289160Z  * [new branch]              gh/XuehaiPan/359/base       -> origin/gh/XuehaiPan/359/base
2025-09-07T07:36:03.7290331Z  * [new branch]              gh/XuehaiPan/359/head       -> origin/gh/XuehaiPan/359/head
2025-09-07T07:36:03.7291562Z  * [new branch]              gh/XuehaiPan/359/orig       -> origin/gh/XuehaiPan/359/orig
2025-09-07T07:36:03.7293222Z  * [new branch]              gh/XuehaiPan/360/base       -> origin/gh/XuehaiPan/360/base
2025-09-07T07:36:03.7294518Z  * [new branch]              gh/XuehaiPan/360/head       -> origin/gh/XuehaiPan/360/head
2025-09-07T07:36:03.7295707Z  * [new branch]              gh/XuehaiPan/360/orig       -> origin/gh/XuehaiPan/360/orig
2025-09-07T07:36:03.7297521Z  * [new branch]              gh/XuehaiPan/365/base       -> origin/gh/XuehaiPan/365/base
2025-09-07T07:36:03.7298810Z  * [new branch]              gh/XuehaiPan/365/head       -> origin/gh/XuehaiPan/365/head
2025-09-07T07:36:03.7299994Z  * [new branch]              gh/XuehaiPan/365/orig       -> origin/gh/XuehaiPan/365/orig
2025-09-07T07:36:03.7301832Z  * [new branch]              gh/XuehaiPan/366/base       -> origin/gh/XuehaiPan/366/base
2025-09-07T07:36:03.7303187Z  * [new branch]              gh/XuehaiPan/366/head       -> origin/gh/XuehaiPan/366/head
2025-09-07T07:36:03.7305077Z  * [new branch]              gh/XuehaiPan/369/base       -> origin/gh/XuehaiPan/369/base
2025-09-07T07:36:03.7306466Z  * [new branch]              gh/XuehaiPan/369/head       -> origin/gh/XuehaiPan/369/head
2025-09-07T07:36:03.7307667Z  * [new branch]              gh/XuehaiPan/369/orig       -> origin/gh/XuehaiPan/369/orig
2025-09-07T07:36:03.7309386Z  * [new branch]              gh/XuehaiPan/370/base       -> origin/gh/XuehaiPan/370/base
2025-09-07T07:36:03.7310696Z  * [new branch]              gh/XuehaiPan/370/head       -> origin/gh/XuehaiPan/370/head
2025-09-07T07:36:03.7311992Z  * [new branch]              gh/XuehaiPan/370/orig       -> origin/gh/XuehaiPan/370/orig
2025-09-07T07:36:03.7313856Z  * [new branch]              gh/XuehaiPan/380/base       -> origin/gh/XuehaiPan/380/base
2025-09-07T07:36:03.7315069Z  * [new branch]              gh/XuehaiPan/380/head       -> origin/gh/XuehaiPan/380/head
2025-09-07T07:36:03.7316402Z  * [new branch]              gh/XuehaiPan/380/orig       -> origin/gh/XuehaiPan/380/orig
2025-09-07T07:36:03.7318032Z  * [new branch]              gh/XuehaiPan/381/base       -> origin/gh/XuehaiPan/381/base
2025-09-07T07:36:03.7319220Z  * [new branch]              gh/XuehaiPan/381/head       -> origin/gh/XuehaiPan/381/head
2025-09-07T07:36:03.7320989Z  * [new branch]              gh/XuehaiPan/382/base       -> origin/gh/XuehaiPan/382/base
2025-09-07T07:36:03.7322237Z  * [new branch]              gh/XuehaiPan/382/head       -> origin/gh/XuehaiPan/382/head
2025-09-07T07:36:03.7323418Z  * [new branch]              gh/XuehaiPan/382/orig       -> origin/gh/XuehaiPan/382/orig
2025-09-07T07:36:03.7325263Z  * [new branch]              gh/XuehaiPan/383/base       -> origin/gh/XuehaiPan/383/base
2025-09-07T07:36:03.7326487Z  * [new branch]              gh/XuehaiPan/383/head       -> origin/gh/XuehaiPan/383/head
2025-09-07T07:36:03.7327691Z  * [new branch]              gh/XuehaiPan/383/orig       -> origin/gh/XuehaiPan/383/orig
2025-09-07T07:36:03.7329560Z  * [new branch]              gh/XuehaiPan/384/base       -> origin/gh/XuehaiPan/384/base
2025-09-07T07:36:03.7330771Z  * [new branch]              gh/XuehaiPan/384/head       -> origin/gh/XuehaiPan/384/head
2025-09-07T07:36:03.7331961Z  * [new branch]              gh/XuehaiPan/384/orig       -> origin/gh/XuehaiPan/384/orig
2025-09-07T07:36:03.7333750Z  * [new branch]              gh/XuehaiPan/385/base       -> origin/gh/XuehaiPan/385/base
2025-09-07T07:36:03.7334965Z  * [new branch]              gh/XuehaiPan/385/head       -> origin/gh/XuehaiPan/385/head
2025-09-07T07:36:03.7336090Z  * [new branch]              gh/XuehaiPan/385/orig       -> origin/gh/XuehaiPan/385/orig
2025-09-07T07:36:03.7337826Z  * [new branch]              gh/XuehaiPan/386/base       -> origin/gh/XuehaiPan/386/base
2025-09-07T07:36:03.7339104Z  * [new branch]              gh/XuehaiPan/386/head       -> origin/gh/XuehaiPan/386/head
2025-09-07T07:36:03.7340277Z  * [new branch]              gh/XuehaiPan/386/orig       -> origin/gh/XuehaiPan/386/orig
2025-09-07T07:36:03.7342130Z  * [new branch]              gh/XuehaiPan/387/base       -> origin/gh/XuehaiPan/387/base
2025-09-07T07:36:03.7343324Z  * [new branch]              gh/XuehaiPan/387/head       -> origin/gh/XuehaiPan/387/head
2025-09-07T07:36:03.7344527Z  * [new branch]              gh/XuehaiPan/387/orig       -> origin/gh/XuehaiPan/387/orig
2025-09-07T07:36:03.7346761Z  * [new branch]              gh/ZainRizvi/1/base         -> origin/gh/ZainRizvi/1/base
2025-09-07T07:36:03.7347952Z  * [new branch]              gh/ZainRizvi/1/head         -> origin/gh/ZainRizvi/1/head
2025-09-07T07:36:03.7349600Z  * [new branch]              gh/ZainRizvi/2/base         -> origin/gh/ZainRizvi/2/base
2025-09-07T07:36:03.7350673Z  * [new branch]              gh/ZainRizvi/2/head         -> origin/gh/ZainRizvi/2/head
2025-09-07T07:36:03.7352334Z  * [new branch]              gh/ZainRizvi/3/base         -> origin/gh/ZainRizvi/3/base
2025-09-07T07:36:03.7353524Z  * [new branch]              gh/ZainRizvi/3/head         -> origin/gh/ZainRizvi/3/head
2025-09-07T07:36:03.7355215Z  * [new branch]              gh/ZainRizvi/4/base         -> origin/gh/ZainRizvi/4/base
2025-09-07T07:36:03.7356425Z  * [new branch]              gh/ZainRizvi/4/head         -> origin/gh/ZainRizvi/4/head
2025-09-07T07:36:03.7358121Z  * [new branch]              gh/ZainRizvi/5/base         -> origin/gh/ZainRizvi/5/base
2025-09-07T07:36:03.7359296Z  * [new branch]              gh/ZainRizvi/5/head         -> origin/gh/ZainRizvi/5/head
2025-09-07T07:36:03.7361015Z  * [new branch]              gh/ZainRizvi/6/base         -> origin/gh/ZainRizvi/6/base
2025-09-07T07:36:03.7362618Z  * [new branch]              gh/ZainRizvi/6/head         -> origin/gh/ZainRizvi/6/head
2025-09-07T07:36:03.7363848Z  * [new branch]              gh/ZainRizvi/6/orig         -> origin/gh/ZainRizvi/6/orig
2025-09-07T07:36:03.7365528Z  * [new branch]              gh/ZainRizvi/7/base         -> origin/gh/ZainRizvi/7/base
2025-09-07T07:36:03.7366799Z  * [new branch]              gh/ZainRizvi/7/head         -> origin/gh/ZainRizvi/7/head
2025-09-07T07:36:03.7367905Z  * [new branch]              gh/ZainRizvi/7/orig         -> origin/gh/ZainRizvi/7/orig
2025-09-07T07:36:03.7369734Z  * [new branch]              gh/ZainRizvi/8/base         -> origin/gh/ZainRizvi/8/base
2025-09-07T07:36:03.7370908Z  * [new branch]              gh/ZainRizvi/8/head         -> origin/gh/ZainRizvi/8/head
2025-09-07T07:36:03.7372588Z  * [new branch]              gh/ZainRizvi/9/base         -> origin/gh/ZainRizvi/9/base
2025-09-07T07:36:03.7373784Z  * [new branch]              gh/ZainRizvi/9/head         -> origin/gh/ZainRizvi/9/head
2025-09-07T07:36:03.7374990Z  * [new branch]              gh/ZainRizvi/9/orig         -> origin/gh/ZainRizvi/9/orig
2025-09-07T07:36:03.7377474Z  * [new branch]              gh/ZhiweiYan-96/39/base     -> origin/gh/ZhiweiYan-96/39/base
2025-09-07T07:36:03.7378562Z  * [new branch]              gh/ZhiweiYan-96/39/head     -> origin/gh/ZhiweiYan-96/39/head
2025-09-07T07:36:03.7379943Z  * [new branch]              gh/ZhiweiYan-96/39/orig     -> origin/gh/ZhiweiYan-96/39/orig
2025-09-07T07:36:03.7381705Z  * [new branch]              gh/ZhiweiYan-96/44/base     -> origin/gh/ZhiweiYan-96/44/base
2025-09-07T07:36:03.7382911Z  * [new branch]              gh/ZhiweiYan-96/44/head     -> origin/gh/ZhiweiYan-96/44/head
2025-09-07T07:36:03.7384576Z  * [new branch]              gh/ZhiweiYan-96/45/base     -> origin/gh/ZhiweiYan-96/45/base
2025-09-07T07:36:03.7385704Z  * [new branch]              gh/ZhiweiYan-96/45/head     -> origin/gh/ZhiweiYan-96/45/head
2025-09-07T07:36:03.7387464Z  * [new branch]              gh/ZhiweiYan-96/49/base     -> origin/gh/ZhiweiYan-96/49/base
2025-09-07T07:36:03.7388649Z  * [new branch]              gh/ZhiweiYan-96/49/head     -> origin/gh/ZhiweiYan-96/49/head
2025-09-07T07:36:03.7390393Z  * [new branch]              gh/ZhiweiYan-96/62/base     -> origin/gh/ZhiweiYan-96/62/base
2025-09-07T07:36:03.7391634Z  * [new branch]              gh/ZhiweiYan-96/62/head     -> origin/gh/ZhiweiYan-96/62/head
2025-09-07T07:36:03.7393413Z  * [new branch]              gh/ZhiweiYan-96/64/base     -> origin/gh/ZhiweiYan-96/64/base
2025-09-07T07:36:03.7394565Z  * [new branch]              gh/ZhiweiYan-96/64/head     -> origin/gh/ZhiweiYan-96/64/head
2025-09-07T07:36:03.7395770Z  * [new branch]              gh/ZhiweiYan-96/64/orig     -> origin/gh/ZhiweiYan-96/64/orig
2025-09-07T07:36:03.7397489Z  * [new branch]              gh/ZhiweiYan-96/65/base     -> origin/gh/ZhiweiYan-96/65/base
2025-09-07T07:36:03.7398674Z  * [new branch]              gh/ZhiweiYan-96/65/head     -> origin/gh/ZhiweiYan-96/65/head
2025-09-07T07:36:03.7399910Z  * [new branch]              gh/ZhiweiYan-96/65/orig     -> origin/gh/ZhiweiYan-96/65/orig
2025-09-07T07:36:03.7401590Z  * [new branch]              gh/ZhiweiYan-96/66/base     -> origin/gh/ZhiweiYan-96/66/base
2025-09-07T07:36:03.7402856Z  * [new branch]              gh/ZhiweiYan-96/66/head     -> origin/gh/ZhiweiYan-96/66/head
2025-09-07T07:36:03.7404753Z  * [new branch]              gh/ZhiweiYan-96/67/base     -> origin/gh/ZhiweiYan-96/67/base
2025-09-07T07:36:03.7405875Z  * [new branch]              gh/ZhiweiYan-96/67/head     -> origin/gh/ZhiweiYan-96/67/head
2025-09-07T07:36:03.7407502Z  * [new branch]              gh/ZhiweiYan-96/68/base     -> origin/gh/ZhiweiYan-96/68/base
2025-09-07T07:36:03.7408698Z  * [new branch]              gh/ZhiweiYan-96/68/head     -> origin/gh/ZhiweiYan-96/68/head
2025-09-07T07:36:03.7409926Z  * [new branch]              gh/ZhiweiYan-96/68/orig     -> origin/gh/ZhiweiYan-96/68/orig
2025-09-07T07:36:03.7412067Z  * [new branch]              gh/aakhundov/1/base         -> origin/gh/aakhundov/1/base
2025-09-07T07:36:03.7413357Z  * [new branch]              gh/aakhundov/1/head         -> origin/gh/aakhundov/1/head
2025-09-07T07:36:03.7414958Z  * [new branch]              gh/aakhundov/2/base         -> origin/gh/aakhundov/2/base
2025-09-07T07:36:03.7416232Z  * [new branch]              gh/aakhundov/2/head         -> origin/gh/aakhundov/2/head
2025-09-07T07:36:03.7418188Z  * [new branch]              gh/aditew01/openblas        -> origin/gh/aditew01/openblas
2025-09-07T07:36:03.7419239Z  * [new branch]              gh/aditew01/sbgemm          -> origin/gh/aditew01/sbgemm
2025-09-07T07:36:03.7420414Z  * [new branch]              gh/aditew01/vecbf16         -> origin/gh/aditew01/vecbf16
2025-09-07T07:36:03.7422331Z  * [new branch]              gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init
2025-09-07T07:36:03.7424412Z  * [new branch]              gh/alexsamardzic/9/base     -> origin/gh/alexsamardzic/9/base
2025-09-07T07:36:03.7425581Z  * [new branch]              gh/alexsamardzic/9/head     -> origin/gh/alexsamardzic/9/head
2025-09-07T07:36:03.7426813Z  * [new branch]              gh/alexsamardzic/9/orig     -> origin/gh/alexsamardzic/9/orig
2025-09-07T07:36:03.7429016Z  * [new branch]              gh/amjames/18/base          -> origin/gh/amjames/18/base
2025-09-07T07:36:03.7430161Z  * [new branch]              gh/amjames/18/head          -> origin/gh/amjames/18/head
2025-09-07T07:36:03.7431352Z  * [new branch]              gh/amjames/18/orig          -> origin/gh/amjames/18/orig
2025-09-07T07:36:03.7433672Z  * [new branch]              gh/andrewor14/35/base       -> origin/gh/andrewor14/35/base
2025-09-07T07:36:03.7434939Z  * [new branch]              gh/andrewor14/35/head       -> origin/gh/andrewor14/35/head
2025-09-07T07:36:03.7436142Z  * [new branch]              gh/andrewor14/35/orig       -> origin/gh/andrewor14/35/orig
2025-09-07T07:36:03.7438008Z  * [new branch]              gh/andrewor14/50/base       -> origin/gh/andrewor14/50/base
2025-09-07T07:36:03.7439391Z  * [new branch]              gh/andrewor14/50/head       -> origin/gh/andrewor14/50/head
2025-09-07T07:36:03.7440641Z  * [new branch]              gh/andrewor14/50/orig       -> origin/gh/andrewor14/50/orig
2025-09-07T07:36:03.7442355Z  * [new branch]              gh/andrewor14/51/base       -> origin/gh/andrewor14/51/base
2025-09-07T07:36:03.7443624Z  * [new branch]              gh/andrewor14/51/orig       -> origin/gh/andrewor14/51/orig
2025-09-07T07:36:03.7445824Z  * [new branch]              gh/andyanwang/1/base        -> origin/gh/andyanwang/1/base
2025-09-07T07:36:03.7447005Z  * [new branch]              gh/andyanwang/1/head        -> origin/gh/andyanwang/1/head
2025-09-07T07:36:03.7448162Z  * [new branch]              gh/andyanwang/1/orig        -> origin/gh/andyanwang/1/orig
2025-09-07T07:36:03.7450083Z  * [new branch]              gh/andyanwang/13/base       -> origin/gh/andyanwang/13/base
2025-09-07T07:36:03.7451302Z  * [new branch]              gh/andyanwang/13/head       -> origin/gh/andyanwang/13/head
2025-09-07T07:36:03.7453034Z  * [new branch]              gh/andyanwang/13/orig       -> origin/gh/andyanwang/13/orig
2025-09-07T07:36:03.7454877Z  * [new branch]              gh/andyanwang/2/base        -> origin/gh/andyanwang/2/base
2025-09-07T07:36:03.7456221Z  * [new branch]              gh/andyanwang/2/head        -> origin/gh/andyanwang/2/head
2025-09-07T07:36:03.7457438Z  * [new branch]              gh/andyanwang/2/orig        -> origin/gh/andyanwang/2/orig
2025-09-07T07:36:03.7459277Z  * [new branch]              gh/andyanwang/28/base       -> origin/gh/andyanwang/28/base
2025-09-07T07:36:03.7460509Z  * [new branch]              gh/andyanwang/28/head       -> origin/gh/andyanwang/28/head
2025-09-07T07:36:03.7461734Z  * [new branch]              gh/andyanwang/28/orig       -> origin/gh/andyanwang/28/orig
2025-09-07T07:36:03.7463325Z  * [new branch]              gh/andyanwang/3/base        -> origin/gh/andyanwang/3/base
2025-09-07T07:36:03.7464550Z  * [new branch]              gh/andyanwang/3/head        -> origin/gh/andyanwang/3/head
2025-09-07T07:36:03.7465766Z  * [new branch]              gh/andyanwang/3/orig        -> origin/gh/andyanwang/3/orig
2025-09-07T07:36:03.7467569Z  * [new branch]              gh/andyanwang/30/base       -> origin/gh/andyanwang/30/base
2025-09-07T07:36:03.7468972Z  * [new branch]              gh/andyanwang/30/orig       -> origin/gh/andyanwang/30/orig
2025-09-07T07:36:03.7470629Z  * [new branch]              gh/andyanwang/31/base       -> origin/gh/andyanwang/31/base
2025-09-07T07:36:03.7471966Z  * [new branch]              gh/andyanwang/31/orig       -> origin/gh/andyanwang/31/orig
2025-09-07T07:36:03.7474042Z  * [new branch]              gh/andyanwang/32/base       -> origin/gh/andyanwang/32/base
2025-09-07T07:36:03.7475228Z  * [new branch]              gh/andyanwang/32/head       -> origin/gh/andyanwang/32/head
2025-09-07T07:36:03.7476542Z  * [new branch]              gh/andyanwang/32/orig       -> origin/gh/andyanwang/32/orig
2025-09-07T07:36:03.7478274Z  * [new branch]              gh/andyanwang/39/base       -> origin/gh/andyanwang/39/base
2025-09-07T07:36:03.7479499Z  * [new branch]              gh/andyanwang/39/head       -> origin/gh/andyanwang/39/head
2025-09-07T07:36:03.7480737Z  * [new branch]              gh/andyanwang/39/orig       -> origin/gh/andyanwang/39/orig
2025-09-07T07:36:03.7482544Z  * [new branch]              gh/andyanwang/4/base        -> origin/gh/andyanwang/4/base
2025-09-07T07:36:03.7483679Z  * [new branch]              gh/andyanwang/4/head        -> origin/gh/andyanwang/4/head
2025-09-07T07:36:03.7484919Z  * [new branch]              gh/andyanwang/4/orig        -> origin/gh/andyanwang/4/orig
2025-09-07T07:36:03.7487009Z  * [new branch]              gh/angelayi/107/base        -> origin/gh/angelayi/107/base
2025-09-07T07:36:03.7488290Z  * [new branch]              gh/angelayi/107/head        -> origin/gh/angelayi/107/head
2025-09-07T07:36:03.7490051Z  * [new branch]              gh/angelayi/111/base        -> origin/gh/angelayi/111/base
2025-09-07T07:36:03.7491378Z  * [new branch]              gh/angelayi/111/head        -> origin/gh/angelayi/111/head
2025-09-07T07:36:03.7492557Z  * [new branch]              gh/angelayi/111/orig        -> origin/gh/angelayi/111/orig
2025-09-07T07:36:03.7494269Z  * [new branch]              gh/angelayi/112/base        -> origin/gh/angelayi/112/base
2025-09-07T07:36:03.7495646Z  * [new branch]              gh/angelayi/112/head        -> origin/gh/angelayi/112/head
2025-09-07T07:36:03.7496878Z  * [new branch]              gh/angelayi/112/orig        -> origin/gh/angelayi/112/orig
2025-09-07T07:36:03.7498801Z  * [new branch]              gh/angelayi/113/base        -> origin/gh/angelayi/113/base
2025-09-07T07:36:03.7499914Z  * [new branch]              gh/angelayi/113/head        -> origin/gh/angelayi/113/head
2025-09-07T07:36:03.7501062Z  * [new branch]              gh/angelayi/113/orig        -> origin/gh/angelayi/113/orig
2025-09-07T07:36:03.7502800Z  * [new branch]              gh/angelayi/114/base        -> origin/gh/angelayi/114/base
2025-09-07T07:36:03.7504186Z  * [new branch]              gh/angelayi/114/head        -> origin/gh/angelayi/114/head
2025-09-07T07:36:03.7505430Z  * [new branch]              gh/angelayi/114/orig        -> origin/gh/angelayi/114/orig
2025-09-07T07:36:03.7507210Z  * [new branch]              gh/angelayi/115/base        -> origin/gh/angelayi/115/base
2025-09-07T07:36:03.7508452Z  * [new branch]              gh/angelayi/115/head        -> origin/gh/angelayi/115/head
2025-09-07T07:36:03.7509621Z  * [new branch]              gh/angelayi/115/orig        -> origin/gh/angelayi/115/orig
2025-09-07T07:36:03.7511827Z  * [new branch]              gh/anijain2305/753/base     -> origin/gh/anijain2305/753/base
2025-09-07T07:36:03.7513021Z  * [new branch]              gh/anijain2305/753/head     -> origin/gh/anijain2305/753/head
2025-09-07T07:36:03.7514205Z  * [new branch]              gh/anijain2305/753/orig     -> origin/gh/anijain2305/753/orig
2025-09-07T07:36:03.7516070Z  * [new branch]              gh/anijain2305/766/base     -> origin/gh/anijain2305/766/base
2025-09-07T07:36:03.7517174Z  * [new branch]              gh/anijain2305/766/head     -> origin/gh/anijain2305/766/head
2025-09-07T07:36:03.7518495Z  * [new branch]              gh/anijain2305/766/orig     -> origin/gh/anijain2305/766/orig
2025-09-07T07:36:03.7520206Z  * [new branch]              gh/anijain2305/790/base     -> origin/gh/anijain2305/790/base
2025-09-07T07:36:03.7521286Z  * [new branch]              gh/anijain2305/790/head     -> origin/gh/anijain2305/790/head
2025-09-07T07:36:03.7522582Z  * [new branch]              gh/anijain2305/790/orig     -> origin/gh/anijain2305/790/orig
2025-09-07T07:36:03.7524246Z  * [new branch]              gh/anijain2305/792/base     -> origin/gh/anijain2305/792/base
2025-09-07T07:36:03.7525473Z  * [new branch]              gh/anijain2305/792/head     -> origin/gh/anijain2305/792/head
2025-09-07T07:36:03.7526843Z  * [new branch]              gh/anijain2305/792/orig     -> origin/gh/anijain2305/792/orig
2025-09-07T07:36:03.7528515Z  * [new branch]              gh/anijain2305/803/base     -> origin/gh/anijain2305/803/base
2025-09-07T07:36:03.7529725Z  * [new branch]              gh/anijain2305/803/head     -> origin/gh/anijain2305/803/head
2025-09-07T07:36:03.7530900Z  * [new branch]              gh/anijain2305/803/orig     -> origin/gh/anijain2305/803/orig
2025-09-07T07:36:03.7532596Z  * [new branch]              gh/anijain2305/804/base     -> origin/gh/anijain2305/804/base
2025-09-07T07:36:03.7533759Z  * [new branch]              gh/anijain2305/804/head     -> origin/gh/anijain2305/804/head
2025-09-07T07:36:03.7534992Z  * [new branch]              gh/anijain2305/804/orig     -> origin/gh/anijain2305/804/orig
2025-09-07T07:36:03.7536683Z  * [new branch]              gh/anijain2305/805/base     -> origin/gh/anijain2305/805/base
2025-09-07T07:36:03.7537944Z  * [new branch]              gh/anijain2305/805/head     -> origin/gh/anijain2305/805/head
2025-09-07T07:36:03.7539212Z  * [new branch]              gh/anijain2305/805/orig     -> origin/gh/anijain2305/805/orig
2025-09-07T07:36:03.7541173Z  * [new branch]              gh/anijain2305/810/base     -> origin/gh/anijain2305/810/base
2025-09-07T07:36:03.7542511Z  * [new branch]              gh/anijain2305/810/head     -> origin/gh/anijain2305/810/head
2025-09-07T07:36:03.7543503Z  * [new branch]              gh/anijain2305/810/orig     -> origin/gh/anijain2305/810/orig
2025-09-07T07:36:03.7545297Z  * [new branch]              gh/anijain2305/812/base     -> origin/gh/anijain2305/812/base
2025-09-07T07:36:03.7546537Z  * [new branch]              gh/anijain2305/812/head     -> origin/gh/anijain2305/812/head
2025-09-07T07:36:03.7547709Z  * [new branch]              gh/anijain2305/812/orig     -> origin/gh/anijain2305/812/orig
2025-09-07T07:36:03.7549461Z  * [new branch]              gh/anijain2305/838/base     -> origin/gh/anijain2305/838/base
2025-09-07T07:36:03.7550657Z  * [new branch]              gh/anijain2305/838/head     -> origin/gh/anijain2305/838/head
2025-09-07T07:36:03.7551871Z  * [new branch]              gh/anijain2305/838/orig     -> origin/gh/anijain2305/838/orig
2025-09-07T07:36:03.7553554Z  * [new branch]              gh/anijain2305/839/base     -> origin/gh/anijain2305/839/base
2025-09-07T07:36:03.7554827Z  * [new branch]              gh/anijain2305/839/head     -> origin/gh/anijain2305/839/head
2025-09-07T07:36:03.7556000Z  * [new branch]              gh/anijain2305/839/orig     -> origin/gh/anijain2305/839/orig
2025-09-07T07:36:03.7557688Z  * [new branch]              gh/anijain2305/843/base     -> origin/gh/anijain2305/843/base
2025-09-07T07:36:03.7558867Z  * [new branch]              gh/anijain2305/843/head     -> origin/gh/anijain2305/843/head
2025-09-07T07:36:03.7560071Z  * [new branch]              gh/anijain2305/843/orig     -> origin/gh/anijain2305/843/orig
2025-09-07T07:36:03.7561949Z  * [new branch]              gh/anijain2305/844/base     -> origin/gh/anijain2305/844/base
2025-09-07T07:36:03.7563130Z  * [new branch]              gh/anijain2305/844/head     -> origin/gh/anijain2305/844/head
2025-09-07T07:36:03.7564352Z  * [new branch]              gh/anijain2305/844/orig     -> origin/gh/anijain2305/844/orig
2025-09-07T07:36:03.7566096Z  * [new branch]              gh/anijain2305/846/base     -> origin/gh/anijain2305/846/base
2025-09-07T07:36:03.7567381Z  * [new branch]              gh/anijain2305/846/head     -> origin/gh/anijain2305/846/head
2025-09-07T07:36:03.7568514Z  * [new branch]              gh/anijain2305/846/orig     -> origin/gh/anijain2305/846/orig
2025-09-07T07:36:03.7570278Z  * [new branch]              gh/anijain2305/848/base     -> origin/gh/anijain2305/848/base
2025-09-07T07:36:03.7571552Z  * [new branch]              gh/anijain2305/848/head     -> origin/gh/anijain2305/848/head
2025-09-07T07:36:03.7572880Z  * [new branch]              gh/anijain2305/848/orig     -> origin/gh/anijain2305/848/orig
2025-09-07T07:36:03.7574599Z  * [new branch]              gh/anijain2305/849/base     -> origin/gh/anijain2305/849/base
2025-09-07T07:36:03.7575825Z  * [new branch]              gh/anijain2305/849/head     -> origin/gh/anijain2305/849/head
2025-09-07T07:36:03.7576996Z  * [new branch]              gh/anijain2305/849/orig     -> origin/gh/anijain2305/849/orig
2025-09-07T07:36:03.7579451Z  * [new branch]              gh/anijain2305/850/base     -> origin/gh/anijain2305/850/base
2025-09-07T07:36:03.7580349Z  * [new branch]              gh/anijain2305/850/head     -> origin/gh/anijain2305/850/head
2025-09-07T07:36:03.7581584Z  * [new branch]              gh/anijain2305/850/orig     -> origin/gh/anijain2305/850/orig
2025-09-07T07:36:03.7583393Z  * [new branch]              gh/anijain2305/851/base     -> origin/gh/anijain2305/851/base
2025-09-07T07:36:03.7584645Z  * [new branch]              gh/anijain2305/851/head     -> origin/gh/anijain2305/851/head
2025-09-07T07:36:03.7585808Z  * [new branch]              gh/anijain2305/851/orig     -> origin/gh/anijain2305/851/orig
2025-09-07T07:36:03.7587670Z  * [new branch]              gh/anijain2305/852/base     -> origin/gh/anijain2305/852/base
2025-09-07T07:36:03.7588908Z  * [new branch]              gh/anijain2305/852/head     -> origin/gh/anijain2305/852/head
2025-09-07T07:36:03.7590184Z  * [new branch]              gh/anijain2305/852/orig     -> origin/gh/anijain2305/852/orig
2025-09-07T07:36:03.7591931Z  * [new branch]              gh/anijain2305/853/base     -> origin/gh/anijain2305/853/base
2025-09-07T07:36:03.7593054Z  * [new branch]              gh/anijain2305/853/head     -> origin/gh/anijain2305/853/head
2025-09-07T07:36:03.7594209Z  * [new branch]              gh/anijain2305/853/orig     -> origin/gh/anijain2305/853/orig
2025-09-07T07:36:03.7595997Z  * [new branch]              gh/anijain2305/854/base     -> origin/gh/anijain2305/854/base
2025-09-07T07:36:03.7597369Z  * [new branch]              gh/anijain2305/854/head     -> origin/gh/anijain2305/854/head
2025-09-07T07:36:03.7598549Z  * [new branch]              gh/anijain2305/854/orig     -> origin/gh/anijain2305/854/orig
2025-09-07T07:36:03.7600376Z  * [new branch]              gh/anijain2305/855/base     -> origin/gh/anijain2305/855/base
2025-09-07T07:36:03.7601559Z  * [new branch]              gh/anijain2305/855/head     -> origin/gh/anijain2305/855/head
2025-09-07T07:36:03.7602798Z  * [new branch]              gh/anijain2305/855/orig     -> origin/gh/anijain2305/855/orig
2025-09-07T07:36:03.7606660Z  * [new branch]              gh/anijain2305/856/base     -> origin/gh/anijain2305/856/base
2025-09-07T07:36:03.7607869Z  * [new branch]              gh/anijain2305/856/head     -> origin/gh/anijain2305/856/head
2025-09-07T07:36:03.7609123Z  * [new branch]              gh/anijain2305/856/orig     -> origin/gh/anijain2305/856/orig
2025-09-07T07:36:03.7610871Z  * [new branch]              gh/anijain2305/857/base     -> origin/gh/anijain2305/857/base
2025-09-07T07:36:03.7612075Z  * [new branch]              gh/anijain2305/857/head     -> origin/gh/anijain2305/857/head
2025-09-07T07:36:03.7613274Z  * [new branch]              gh/anijain2305/857/orig     -> origin/gh/anijain2305/857/orig
2025-09-07T07:36:03.7614993Z  * [new branch]              gh/anijain2305/858/base     -> origin/gh/anijain2305/858/base
2025-09-07T07:36:03.7616179Z  * [new branch]              gh/anijain2305/858/head     -> origin/gh/anijain2305/858/head
2025-09-07T07:36:03.7617472Z  * [new branch]              gh/anijain2305/858/orig     -> origin/gh/anijain2305/858/orig
2025-09-07T07:36:03.7619288Z  * [new branch]              gh/anijain2305/859/base     -> origin/gh/anijain2305/859/base
2025-09-07T07:36:03.7620490Z  * [new branch]              gh/anijain2305/859/head     -> origin/gh/anijain2305/859/head
2025-09-07T07:36:03.7621684Z  * [new branch]              gh/anijain2305/859/orig     -> origin/gh/anijain2305/859/orig
2025-09-07T07:36:03.7623427Z  * [new branch]              gh/anijain2305/860/base     -> origin/gh/anijain2305/860/base
2025-09-07T07:36:03.7624651Z  * [new branch]              gh/anijain2305/860/head     -> origin/gh/anijain2305/860/head
2025-09-07T07:36:03.7625905Z  * [new branch]              gh/anijain2305/860/orig     -> origin/gh/anijain2305/860/orig
2025-09-07T07:36:03.7627679Z  * [new branch]              gh/anijain2305/861/base     -> origin/gh/anijain2305/861/base
2025-09-07T07:36:03.7628858Z  * [new branch]              gh/anijain2305/861/head     -> origin/gh/anijain2305/861/head
2025-09-07T07:36:03.7630129Z  * [new branch]              gh/anijain2305/861/orig     -> origin/gh/anijain2305/861/orig
2025-09-07T07:36:03.7631913Z  * [new branch]              gh/anijain2305/862/base     -> origin/gh/anijain2305/862/base
2025-09-07T07:36:03.7633181Z  * [new branch]              gh/anijain2305/862/head     -> origin/gh/anijain2305/862/head
2025-09-07T07:36:03.7634426Z  * [new branch]              gh/anijain2305/862/orig     -> origin/gh/anijain2305/862/orig
2025-09-07T07:36:03.7636324Z  * [new branch]              gh/anijain2305/863/base     -> origin/gh/anijain2305/863/base
2025-09-07T07:36:03.7637560Z  * [new branch]              gh/anijain2305/863/head     -> origin/gh/anijain2305/863/head
2025-09-07T07:36:03.7638793Z  * [new branch]              gh/anijain2305/863/orig     -> origin/gh/anijain2305/863/orig
2025-09-07T07:36:03.7640605Z  * [new branch]              gh/anijain2305/864/base     -> origin/gh/anijain2305/864/base
2025-09-07T07:36:03.7641790Z  * [new branch]              gh/anijain2305/864/head     -> origin/gh/anijain2305/864/head
2025-09-07T07:36:03.7643029Z  * [new branch]              gh/anijain2305/864/orig     -> origin/gh/anijain2305/864/orig
2025-09-07T07:36:03.7644809Z  * [new branch]              gh/anijain2305/865/base     -> origin/gh/anijain2305/865/base
2025-09-07T07:36:03.7646003Z  * [new branch]              gh/anijain2305/865/head     -> origin/gh/anijain2305/865/head
2025-09-07T07:36:03.7647155Z  * [new branch]              gh/anijain2305/865/orig     -> origin/gh/anijain2305/865/orig
2025-09-07T07:36:03.7648932Z  * [new branch]              gh/anijain2305/866/base     -> origin/gh/anijain2305/866/base
2025-09-07T07:36:03.7650123Z  * [new branch]              gh/anijain2305/866/head     -> origin/gh/anijain2305/866/head
2025-09-07T07:36:03.7651322Z  * [new branch]              gh/anijain2305/866/orig     -> origin/gh/anijain2305/866/orig
2025-09-07T07:36:03.7653584Z  * [new branch]              gh/anjali411/216/base       -> origin/gh/anjali411/216/base
2025-09-07T07:36:03.7654739Z  * [new branch]              gh/anjali411/216/head       -> origin/gh/anjali411/216/head
2025-09-07T07:36:03.7655928Z  * [new branch]              gh/anjali411/216/orig       -> origin/gh/anjali411/216/orig
2025-09-07T07:36:03.7658286Z  * [new branch]              gh/ankitageorge/13/base     -> origin/gh/ankitageorge/13/base
2025-09-07T07:36:03.7659512Z  * [new branch]              gh/ankitageorge/13/head     -> origin/gh/ankitageorge/13/head
2025-09-07T07:36:03.7660756Z  * [new branch]              gh/ankitageorge/13/orig     -> origin/gh/ankitageorge/13/orig
2025-09-07T07:36:03.7662585Z  * [new branch]              gh/ankitageorge/14/base     -> origin/gh/ankitageorge/14/base
2025-09-07T07:36:03.7663717Z  * [new branch]              gh/ankitageorge/14/head     -> origin/gh/ankitageorge/14/head
2025-09-07T07:36:03.7665077Z  * [new branch]              gh/ankitageorge/14/orig     -> origin/gh/ankitageorge/14/orig
2025-09-07T07:36:03.7666922Z  * [new branch]              gh/ankitageorge/15/base     -> origin/gh/ankitageorge/15/base
2025-09-07T07:36:03.7668114Z  * [new branch]              gh/ankitageorge/15/head     -> origin/gh/ankitageorge/15/head
2025-09-07T07:36:03.7669377Z  * [new branch]              gh/ankitageorge/15/orig     -> origin/gh/ankitageorge/15/orig
2025-09-07T07:36:03.7671177Z  * [new branch]              gh/ankitageorge/16/base     -> origin/gh/ankitageorge/16/base
2025-09-07T07:36:03.7672554Z  * [new branch]              gh/ankitageorge/16/head     -> origin/gh/ankitageorge/16/head
2025-09-07T07:36:03.7673822Z  * [new branch]              gh/ankitageorge/16/orig     -> origin/gh/ankitageorge/16/orig
2025-09-07T07:36:03.7675836Z  * [new branch]              gh/ankitageorge/17/base     -> origin/gh/ankitageorge/17/base
2025-09-07T07:36:03.7677020Z  * [new branch]              gh/ankitageorge/17/head     -> origin/gh/ankitageorge/17/head
2025-09-07T07:36:03.7678238Z  * [new branch]              gh/ankitageorge/17/orig     -> origin/gh/ankitageorge/17/orig
2025-09-07T07:36:03.7680098Z  * [new branch]              gh/ankitageorge/21/base     -> origin/gh/ankitageorge/21/base
2025-09-07T07:36:03.7681224Z  * [new branch]              gh/ankitageorge/21/head     -> origin/gh/ankitageorge/21/head
2025-09-07T07:36:03.7682406Z  * [new branch]              gh/ankitageorge/21/orig     -> origin/gh/ankitageorge/21/orig
2025-09-07T07:36:03.7684712Z  * [new branch]              gh/anshul-si/1/base         -> origin/gh/anshul-si/1/base
2025-09-07T07:36:03.7685911Z  * [new branch]              gh/anshul-si/1/head         -> origin/gh/anshul-si/1/head
2025-09-07T07:36:03.7687626Z  * [new branch]              gh/anshul-si/15/base        -> origin/gh/anshul-si/15/base
2025-09-07T07:36:03.7688831Z  * [new branch]              gh/anshul-si/15/head        -> origin/gh/anshul-si/15/head
2025-09-07T07:36:03.7690019Z  * [new branch]              gh/anshul-si/15/orig        -> origin/gh/anshul-si/15/orig
2025-09-07T07:36:03.7691943Z  * [new branch]              gh/anshul-si/16/base        -> origin/gh/anshul-si/16/base
2025-09-07T07:36:03.7693101Z  * [new branch]              gh/anshul-si/16/head        -> origin/gh/anshul-si/16/head
2025-09-07T07:36:03.7694334Z  * [new branch]              gh/anshul-si/16/orig        -> origin/gh/anshul-si/16/orig
2025-09-07T07:36:03.7696192Z  * [new branch]              gh/anshul-si/17/base        -> origin/gh/anshul-si/17/base
2025-09-07T07:36:03.7697450Z  * [new branch]              gh/anshul-si/17/head        -> origin/gh/anshul-si/17/head
2025-09-07T07:36:03.7699101Z  * [new branch]              gh/anshul-si/17/orig        -> origin/gh/anshul-si/17/orig
2025-09-07T07:36:03.7700772Z  * [new branch]              gh/anshul-si/18/base        -> origin/gh/anshul-si/18/base
2025-09-07T07:36:03.7701952Z  * [new branch]              gh/anshul-si/18/head        -> origin/gh/anshul-si/18/head
2025-09-07T07:36:03.7703400Z  * [new branch]              gh/anshul-si/18/orig        -> origin/gh/anshul-si/18/orig
2025-09-07T07:36:03.7705384Z  * [new branch]              gh/anshul-si/19/base        -> origin/gh/anshul-si/19/base
2025-09-07T07:36:03.7706457Z  * [new branch]              gh/anshul-si/19/head        -> origin/gh/anshul-si/19/head
2025-09-07T07:36:03.7707692Z  * [new branch]              gh/anshul-si/19/orig        -> origin/gh/anshul-si/19/orig
2025-09-07T07:36:03.7709511Z  * [new branch]              gh/anshul-si/2/base         -> origin/gh/anshul-si/2/base
2025-09-07T07:36:03.7710680Z  * [new branch]              gh/anshul-si/2/head         -> origin/gh/anshul-si/2/head
2025-09-07T07:36:03.7712607Z  * [new branch]              gh/anshul-si/20/base        -> origin/gh/anshul-si/20/base
2025-09-07T07:36:03.7713815Z  * [new branch]              gh/anshul-si/20/head        -> origin/gh/anshul-si/20/head
2025-09-07T07:36:03.7714995Z  * [new branch]              gh/anshul-si/20/orig        -> origin/gh/anshul-si/20/orig
2025-09-07T07:36:03.7716677Z  * [new branch]              gh/anshul-si/21/base        -> origin/gh/anshul-si/21/base
2025-09-07T07:36:03.7718026Z  * [new branch]              gh/anshul-si/21/head        -> origin/gh/anshul-si/21/head
2025-09-07T07:36:03.7719126Z  * [new branch]              gh/anshul-si/21/orig        -> origin/gh/anshul-si/21/orig
2025-09-07T07:36:03.7720943Z  * [new branch]              gh/anshul-si/22/base        -> origin/gh/anshul-si/22/base
2025-09-07T07:36:03.7722177Z  * [new branch]              gh/anshul-si/22/head        -> origin/gh/anshul-si/22/head
2025-09-07T07:36:03.7723392Z  * [new branch]              gh/anshul-si/22/orig        -> origin/gh/anshul-si/22/orig
2025-09-07T07:36:03.7725028Z  * [new branch]              gh/anshul-si/23/base        -> origin/gh/anshul-si/23/base
2025-09-07T07:36:03.7726295Z  * [new branch]              gh/anshul-si/23/head        -> origin/gh/anshul-si/23/head
2025-09-07T07:36:03.7727506Z  * [new branch]              gh/anshul-si/23/orig        -> origin/gh/anshul-si/23/orig
2025-09-07T07:36:03.7729237Z  * [new branch]              gh/anshul-si/24/base        -> origin/gh/anshul-si/24/base
2025-09-07T07:36:03.7730501Z  * [new branch]              gh/anshul-si/24/head        -> origin/gh/anshul-si/24/head
2025-09-07T07:36:03.7731691Z  * [new branch]              gh/anshul-si/24/orig        -> origin/gh/anshul-si/24/orig
2025-09-07T07:36:03.7733450Z  * [new branch]              gh/anshul-si/25/base        -> origin/gh/anshul-si/25/base
2025-09-07T07:36:03.7734678Z  * [new branch]              gh/anshul-si/25/head        -> origin/gh/anshul-si/25/head
2025-09-07T07:36:03.7735875Z  * [new branch]              gh/anshul-si/25/orig        -> origin/gh/anshul-si/25/orig
2025-09-07T07:36:03.7737655Z  * [new branch]              gh/anshul-si/26/base        -> origin/gh/anshul-si/26/base
2025-09-07T07:36:03.7738893Z  * [new branch]              gh/anshul-si/26/head        -> origin/gh/anshul-si/26/head
2025-09-07T07:36:03.7740061Z  * [new branch]              gh/anshul-si/26/orig        -> origin/gh/anshul-si/26/orig
2025-09-07T07:36:03.7741819Z  * [new branch]              gh/anshul-si/27/base        -> origin/gh/anshul-si/27/base
2025-09-07T07:36:03.7743095Z  * [new branch]              gh/anshul-si/27/head        -> origin/gh/anshul-si/27/head
2025-09-07T07:36:03.7744842Z  * [new branch]              gh/anshul-si/27/orig        -> origin/gh/anshul-si/27/orig
2025-09-07T07:36:03.7746483Z  * [new branch]              gh/anshul-si/28/base        -> origin/gh/anshul-si/28/base
2025-09-07T07:36:03.7747662Z  * [new branch]              gh/anshul-si/28/head        -> origin/gh/anshul-si/28/head
2025-09-07T07:36:03.7748861Z  * [new branch]              gh/anshul-si/28/orig        -> origin/gh/anshul-si/28/orig
2025-09-07T07:36:03.7750456Z  * [new branch]              gh/anshul-si/29/base        -> origin/gh/anshul-si/29/base
2025-09-07T07:36:03.7751807Z  * [new branch]              gh/anshul-si/29/head        -> origin/gh/anshul-si/29/head
2025-09-07T07:36:03.7753006Z  * [new branch]              gh/anshul-si/29/orig        -> origin/gh/anshul-si/29/orig
2025-09-07T07:36:03.7754670Z  * [new branch]              gh/anshul-si/3/base         -> origin/gh/anshul-si/3/base
2025-09-07T07:36:03.7755793Z  * [new branch]              gh/anshul-si/3/head         -> origin/gh/anshul-si/3/head
2025-09-07T07:36:03.7757432Z  * [new branch]              gh/anshul-si/4/base         -> origin/gh/anshul-si/4/base
2025-09-07T07:36:03.7758536Z  * [new branch]              gh/anshul-si/4/head         -> origin/gh/anshul-si/4/head
2025-09-07T07:36:03.7760598Z  * [new branch]              gh/anshul-si/5/base         -> origin/gh/anshul-si/5/base
2025-09-07T07:36:03.7761813Z  * [new branch]              gh/anshul-si/5/head         -> origin/gh/anshul-si/5/head
2025-09-07T07:36:03.7764157Z  * [new branch]              gh/aorenste/132/base        -> origin/gh/aorenste/132/base
2025-09-07T07:36:03.7765450Z  * [new branch]              gh/aorenste/132/head        -> origin/gh/aorenste/132/head
2025-09-07T07:36:03.7767637Z  * [new branch]              gh/bdhirsh/650/base         -> origin/gh/bdhirsh/650/base
2025-09-07T07:36:03.7769047Z  * [new branch]              gh/bdhirsh/650/head         -> origin/gh/bdhirsh/650/head
2025-09-07T07:36:03.7770178Z  * [new branch]              gh/bdhirsh/650/orig         -> origin/gh/bdhirsh/650/orig
2025-09-07T07:36:03.7771935Z  * [new branch]              gh/bdhirsh/663/base         -> origin/gh/bdhirsh/663/base
2025-09-07T07:36:03.7773174Z  * [new branch]              gh/bdhirsh/663/head         -> origin/gh/bdhirsh/663/head
2025-09-07T07:36:03.7774351Z  * [new branch]              gh/bdhirsh/663/orig         -> origin/gh/bdhirsh/663/orig
2025-09-07T07:36:03.7776231Z  * [new branch]              gh/bdhirsh/665/base         -> origin/gh/bdhirsh/665/base
2025-09-07T07:36:03.7777434Z  * [new branch]              gh/bdhirsh/665/head         -> origin/gh/bdhirsh/665/head
2025-09-07T07:36:03.7778752Z  * [new branch]              gh/bdhirsh/665/orig         -> origin/gh/bdhirsh/665/orig
2025-09-07T07:36:03.7780696Z  * [new branch]              gh/bdhirsh/666/base         -> origin/gh/bdhirsh/666/base
2025-09-07T07:36:03.7782121Z  * [new branch]              gh/bdhirsh/666/head         -> origin/gh/bdhirsh/666/head
2025-09-07T07:36:03.7783340Z  * [new branch]              gh/bdhirsh/666/orig         -> origin/gh/bdhirsh/666/orig
2025-09-07T07:36:03.7785312Z  * [new branch]              gh/bdhirsh/667/base         -> origin/gh/bdhirsh/667/base
2025-09-07T07:36:03.7786464Z  * [new branch]              gh/bdhirsh/667/head         -> origin/gh/bdhirsh/667/head
2025-09-07T07:36:03.7787720Z  * [new branch]              gh/bdhirsh/667/orig         -> origin/gh/bdhirsh/667/orig
2025-09-07T07:36:03.7789396Z  * [new branch]              gh/bdhirsh/668/base         -> origin/gh/bdhirsh/668/base
2025-09-07T07:36:03.7790635Z  * [new branch]              gh/bdhirsh/668/head         -> origin/gh/bdhirsh/668/head
2025-09-07T07:36:03.7791812Z  * [new branch]              gh/bdhirsh/668/orig         -> origin/gh/bdhirsh/668/orig
2025-09-07T07:36:03.7793737Z  * [new branch]              gh/bdhirsh/669/base         -> origin/gh/bdhirsh/669/base
2025-09-07T07:36:03.7808415Z  * [new branch]              gh/bdhirsh/669/head         -> origin/gh/bdhirsh/669/head
2025-09-07T07:36:03.7808785Z  * [new branch]              gh/bdhirsh/669/orig         -> origin/gh/bdhirsh/669/orig
2025-09-07T07:36:03.7808943Z  * [new branch]              gh/bdhirsh/670/base         -> origin/gh/bdhirsh/670/base
2025-09-07T07:36:03.7809085Z  * [new branch]              gh/bdhirsh/670/head         -> origin/gh/bdhirsh/670/head
2025-09-07T07:36:03.7809216Z  * [new branch]              gh/bdhirsh/670/orig         -> origin/gh/bdhirsh/670/orig
2025-09-07T07:36:03.7809387Z  * [new branch]              gh/benjaminglass1/100/base  -> origin/gh/benjaminglass1/100/base
2025-09-07T07:36:03.7809541Z  * [new branch]              gh/benjaminglass1/100/head  -> origin/gh/benjaminglass1/100/head
2025-09-07T07:36:03.7809698Z  * [new branch]              gh/benjaminglass1/100/orig  -> origin/gh/benjaminglass1/100/orig
2025-09-07T07:36:03.7809854Z  * [new branch]              gh/benjaminglass1/101/base  -> origin/gh/benjaminglass1/101/base
2025-09-07T07:36:03.7810007Z  * [new branch]              gh/benjaminglass1/101/head  -> origin/gh/benjaminglass1/101/head
2025-09-07T07:36:03.7810178Z  * [new branch]              gh/benjaminglass1/101/orig  -> origin/gh/benjaminglass1/101/orig
2025-09-07T07:36:03.7811479Z  * [new branch]              gh/benjaminglass1/102/base  -> origin/gh/benjaminglass1/102/base
2025-09-07T07:36:03.7812654Z  * [new branch]              gh/benjaminglass1/102/head  -> origin/gh/benjaminglass1/102/head
2025-09-07T07:36:03.7813844Z  * [new branch]              gh/benjaminglass1/102/orig  -> origin/gh/benjaminglass1/102/orig
2025-09-07T07:36:03.7815601Z  * [new branch]              gh/benjaminglass1/103/base  -> origin/gh/benjaminglass1/103/base
2025-09-07T07:36:03.7816772Z  * [new branch]              gh/benjaminglass1/103/head  -> origin/gh/benjaminglass1/103/head
2025-09-07T07:36:03.7818198Z  * [new branch]              gh/benjaminglass1/103/orig  -> origin/gh/benjaminglass1/103/orig
2025-09-07T07:36:03.7820087Z  * [new branch]              gh/benjaminglass1/104/base  -> origin/gh/benjaminglass1/104/base
2025-09-07T07:36:03.7821150Z  * [new branch]              gh/benjaminglass1/104/head  -> origin/gh/benjaminglass1/104/head
2025-09-07T07:36:03.7822347Z  * [new branch]              gh/benjaminglass1/104/orig  -> origin/gh/benjaminglass1/104/orig
2025-09-07T07:36:03.7824034Z  * [new branch]              gh/benjaminglass1/105/base  -> origin/gh/benjaminglass1/105/base
2025-09-07T07:36:03.7825220Z  * [new branch]              gh/benjaminglass1/105/head  -> origin/gh/benjaminglass1/105/head
2025-09-07T07:36:03.7826419Z  * [new branch]              gh/benjaminglass1/105/orig  -> origin/gh/benjaminglass1/105/orig
2025-09-07T07:36:03.7828249Z  * [new branch]              gh/benjaminglass1/106/base  -> origin/gh/benjaminglass1/106/base
2025-09-07T07:36:03.7829445Z  * [new branch]              gh/benjaminglass1/106/head  -> origin/gh/benjaminglass1/106/head
2025-09-07T07:36:03.7830689Z  * [new branch]              gh/benjaminglass1/106/orig  -> origin/gh/benjaminglass1/106/orig
2025-09-07T07:36:03.7832360Z  * [new branch]              gh/benjaminglass1/79/base   -> origin/gh/benjaminglass1/79/base
2025-09-07T07:36:03.7833535Z  * [new branch]              gh/benjaminglass1/79/head   -> origin/gh/benjaminglass1/79/head
2025-09-07T07:36:03.7834689Z  * [new branch]              gh/benjaminglass1/79/orig   -> origin/gh/benjaminglass1/79/orig
2025-09-07T07:36:03.7836384Z  * [new branch]              gh/benjaminglass1/86/base   -> origin/gh/benjaminglass1/86/base
2025-09-07T07:36:03.7837571Z  * [new branch]              gh/benjaminglass1/86/head   -> origin/gh/benjaminglass1/86/head
2025-09-07T07:36:03.7838863Z  * [new branch]              gh/benjaminglass1/86/orig   -> origin/gh/benjaminglass1/86/orig
2025-09-07T07:36:03.7840567Z  * [new branch]              gh/benjaminglass1/89/base   -> origin/gh/benjaminglass1/89/base
2025-09-07T07:36:03.7841768Z  * [new branch]              gh/benjaminglass1/89/head   -> origin/gh/benjaminglass1/89/head
2025-09-07T07:36:03.7842923Z  * [new branch]              gh/benjaminglass1/89/orig   -> origin/gh/benjaminglass1/89/orig
2025-09-07T07:36:03.7844673Z  * [new branch]              gh/benjaminglass1/91/base   -> origin/gh/benjaminglass1/91/base
2025-09-07T07:36:03.7845891Z  * [new branch]              gh/benjaminglass1/91/head   -> origin/gh/benjaminglass1/91/head
2025-09-07T07:36:03.7847081Z  * [new branch]              gh/benjaminglass1/91/orig   -> origin/gh/benjaminglass1/91/orig
2025-09-07T07:36:03.7848820Z  * [new branch]              gh/benjaminglass1/93/base   -> origin/gh/benjaminglass1/93/base
2025-09-07T07:36:03.7850014Z  * [new branch]              gh/benjaminglass1/93/head   -> origin/gh/benjaminglass1/93/head
2025-09-07T07:36:03.7851269Z  * [new branch]              gh/benjaminglass1/93/orig   -> origin/gh/benjaminglass1/93/orig
2025-09-07T07:36:03.7852937Z  * [new branch]              gh/benjaminglass1/95/base   -> origin/gh/benjaminglass1/95/base
2025-09-07T07:36:03.7854320Z  * [new branch]              gh/benjaminglass1/95/head   -> origin/gh/benjaminglass1/95/head
2025-09-07T07:36:03.7855513Z  * [new branch]              gh/benjaminglass1/95/orig   -> origin/gh/benjaminglass1/95/orig
2025-09-07T07:36:03.7857292Z  * [new branch]              gh/benjaminglass1/97/base   -> origin/gh/benjaminglass1/97/base
2025-09-07T07:36:03.7858564Z  * [new branch]              gh/benjaminglass1/97/head   -> origin/gh/benjaminglass1/97/head
2025-09-07T07:36:03.7859779Z  * [new branch]              gh/benjaminglass1/97/orig   -> origin/gh/benjaminglass1/97/orig
2025-09-07T07:36:03.7861673Z  * [new branch]              gh/benjaminglass1/99/base   -> origin/gh/benjaminglass1/99/base
2025-09-07T07:36:03.7862850Z  * [new branch]              gh/benjaminglass1/99/head   -> origin/gh/benjaminglass1/99/head
2025-09-07T07:36:03.7864179Z  * [new branch]              gh/benjaminglass1/99/orig   -> origin/gh/benjaminglass1/99/orig
2025-09-07T07:36:03.7867352Z  * [new branch]              gh/bobrenjc93/514/base      -> origin/gh/bobrenjc93/514/base
2025-09-07T07:36:03.7867918Z  * [new branch]              gh/bobrenjc93/514/head      -> origin/gh/bobrenjc93/514/head
2025-09-07T07:36:03.7868792Z  * [new branch]              gh/bobrenjc93/514/orig      -> origin/gh/bobrenjc93/514/orig
2025-09-07T07:36:03.7870439Z  * [new branch]              gh/bobrenjc93/521/base      -> origin/gh/bobrenjc93/521/base
2025-09-07T07:36:03.7871624Z  * [new branch]              gh/bobrenjc93/521/head      -> origin/gh/bobrenjc93/521/head
2025-09-07T07:36:03.7872841Z  * [new branch]              gh/bobrenjc93/521/orig      -> origin/gh/bobrenjc93/521/orig
2025-09-07T07:36:03.7874525Z  * [new branch]              gh/bobrenjc93/522/base      -> origin/gh/bobrenjc93/522/base
2025-09-07T07:36:03.7875774Z  * [new branch]              gh/bobrenjc93/522/head      -> origin/gh/bobrenjc93/522/head
2025-09-07T07:36:03.7877025Z  * [new branch]              gh/bobrenjc93/522/orig      -> origin/gh/bobrenjc93/522/orig
2025-09-07T07:36:03.7878712Z  * [new branch]              gh/bobrenjc93/525/base      -> origin/gh/bobrenjc93/525/base
2025-09-07T07:36:03.7879890Z  * [new branch]              gh/bobrenjc93/525/head      -> origin/gh/bobrenjc93/525/head
2025-09-07T07:36:03.7881085Z  * [new branch]              gh/bobrenjc93/525/orig      -> origin/gh/bobrenjc93/525/orig
2025-09-07T07:36:03.7882812Z  * [new branch]              gh/bobrenjc93/526/base      -> origin/gh/bobrenjc93/526/base
2025-09-07T07:36:03.7883991Z  * [new branch]              gh/bobrenjc93/526/head      -> origin/gh/bobrenjc93/526/head
2025-09-07T07:36:03.7885179Z  * [new branch]              gh/bobrenjc93/526/orig      -> origin/gh/bobrenjc93/526/orig
2025-09-07T07:36:03.7886890Z  * [new branch]              gh/bobrenjc93/527/base      -> origin/gh/bobrenjc93/527/base
2025-09-07T07:36:03.7888107Z  * [new branch]              gh/bobrenjc93/527/head      -> origin/gh/bobrenjc93/527/head
2025-09-07T07:36:03.7889451Z  * [new branch]              gh/bobrenjc93/527/orig      -> origin/gh/bobrenjc93/527/orig
2025-09-07T07:36:03.7891153Z  * [new branch]              gh/bobrenjc93/528/base      -> origin/gh/bobrenjc93/528/base
2025-09-07T07:36:03.7892376Z  * [new branch]              gh/bobrenjc93/528/head      -> origin/gh/bobrenjc93/528/head
2025-09-07T07:36:03.7893610Z  * [new branch]              gh/bobrenjc93/528/orig      -> origin/gh/bobrenjc93/528/orig
2025-09-07T07:36:03.7895349Z  * [new branch]              gh/bobrenjc93/529/base      -> origin/gh/bobrenjc93/529/base
2025-09-07T07:36:03.7896507Z  * [new branch]              gh/bobrenjc93/529/head      -> origin/gh/bobrenjc93/529/head
2025-09-07T07:36:03.7897872Z  * [new branch]              gh/bobrenjc93/529/orig      -> origin/gh/bobrenjc93/529/orig
2025-09-07T07:36:03.7899614Z  * [new branch]              gh/bobrenjc93/535/base      -> origin/gh/bobrenjc93/535/base
2025-09-07T07:36:03.7900849Z  * [new branch]              gh/bobrenjc93/535/head      -> origin/gh/bobrenjc93/535/head
2025-09-07T07:36:03.7902020Z  * [new branch]              gh/bobrenjc93/535/orig      -> origin/gh/bobrenjc93/535/orig
2025-09-07T07:36:03.7904106Z  * [new branch]              gh/bobrenjc93/537/base      -> origin/gh/bobrenjc93/537/base
2025-09-07T07:36:03.7905439Z  * [new branch]              gh/bobrenjc93/537/head      -> origin/gh/bobrenjc93/537/head
2025-09-07T07:36:03.7906824Z  * [new branch]              gh/bobrenjc93/537/orig      -> origin/gh/bobrenjc93/537/orig
2025-09-07T07:36:03.7908766Z  * [new branch]              gh/bobrenjc93/539/base      -> origin/gh/bobrenjc93/539/base
2025-09-07T07:36:03.7910002Z  * [new branch]              gh/bobrenjc93/539/head      -> origin/gh/bobrenjc93/539/head
2025-09-07T07:36:03.7911319Z  * [new branch]              gh/bobrenjc93/539/orig      -> origin/gh/bobrenjc93/539/orig
2025-09-07T07:36:03.7913087Z  * [new branch]              gh/bobrenjc93/540/base      -> origin/gh/bobrenjc93/540/base
2025-09-07T07:36:03.7914344Z  * [new branch]              gh/bobrenjc93/540/head      -> origin/gh/bobrenjc93/540/head
2025-09-07T07:36:03.7915713Z  * [new branch]              gh/bobrenjc93/540/orig      -> origin/gh/bobrenjc93/540/orig
2025-09-07T07:36:03.7917344Z  * [new branch]              gh/bobrenjc93/541/base      -> origin/gh/bobrenjc93/541/base
2025-09-07T07:36:03.7918548Z  * [new branch]              gh/bobrenjc93/541/head      -> origin/gh/bobrenjc93/541/head
2025-09-07T07:36:03.7919804Z  * [new branch]              gh/bobrenjc93/541/orig      -> origin/gh/bobrenjc93/541/orig
2025-09-07T07:36:03.7921408Z  * [new branch]              gh/bobrenjc93/542/base      -> origin/gh/bobrenjc93/542/base
2025-09-07T07:36:03.7922571Z  * [new branch]              gh/bobrenjc93/542/head      -> origin/gh/bobrenjc93/542/head
2025-09-07T07:36:03.7923780Z  * [new branch]              gh/bobrenjc93/542/orig      -> origin/gh/bobrenjc93/542/orig
2025-09-07T07:36:03.7925597Z  * [new branch]              gh/bobrenjc93/543/base      -> origin/gh/bobrenjc93/543/base
2025-09-07T07:36:03.7926789Z  * [new branch]              gh/bobrenjc93/543/head      -> origin/gh/bobrenjc93/543/head
2025-09-07T07:36:03.7928057Z  * [new branch]              gh/bobrenjc93/543/orig      -> origin/gh/bobrenjc93/543/orig
2025-09-07T07:36:03.7929684Z  * [new branch]              gh/bobrenjc93/544/base      -> origin/gh/bobrenjc93/544/base
2025-09-07T07:36:03.7930881Z  * [new branch]              gh/bobrenjc93/544/head      -> origin/gh/bobrenjc93/544/head
2025-09-07T07:36:03.7932104Z  * [new branch]              gh/bobrenjc93/544/orig      -> origin/gh/bobrenjc93/544/orig
2025-09-07T07:36:03.7934192Z  * [new branch]              gh/bobrenjc93/545/base      -> origin/gh/bobrenjc93/545/base
2025-09-07T07:36:03.7935584Z  * [new branch]              gh/bobrenjc93/545/head      -> origin/gh/bobrenjc93/545/head
2025-09-07T07:36:03.7936768Z  * [new branch]              gh/bobrenjc93/545/orig      -> origin/gh/bobrenjc93/545/orig
2025-09-07T07:36:03.7938641Z  * [new branch]              gh/bobrenjc93/546/base      -> origin/gh/bobrenjc93/546/base
2025-09-07T07:36:03.7939841Z  * [new branch]              gh/bobrenjc93/546/head      -> origin/gh/bobrenjc93/546/head
2025-09-07T07:36:03.7941029Z  * [new branch]              gh/bobrenjc93/546/orig      -> origin/gh/bobrenjc93/546/orig
2025-09-07T07:36:03.7943315Z  * [new branch]              gh/bobrenjc93/547/base      -> origin/gh/bobrenjc93/547/base
2025-09-07T07:36:03.7944563Z  * [new branch]              gh/bobrenjc93/547/head      -> origin/gh/bobrenjc93/547/head
2025-09-07T07:36:03.7945867Z  * [new branch]              gh/bobrenjc93/547/orig      -> origin/gh/bobrenjc93/547/orig
2025-09-07T07:36:03.7947516Z  * [new branch]              gh/bobrenjc93/548/base      -> origin/gh/bobrenjc93/548/base
2025-09-07T07:36:03.7948695Z  * [new branch]              gh/bobrenjc93/548/head      -> origin/gh/bobrenjc93/548/head
2025-09-07T07:36:03.7949865Z  * [new branch]              gh/bobrenjc93/548/orig      -> origin/gh/bobrenjc93/548/orig
2025-09-07T07:36:03.7951512Z  * [new branch]              gh/bobrenjc93/549/base      -> origin/gh/bobrenjc93/549/base
2025-09-07T07:36:03.7952759Z  * [new branch]              gh/bobrenjc93/549/head      -> origin/gh/bobrenjc93/549/head
2025-09-07T07:36:03.7954022Z  * [new branch]              gh/bobrenjc93/549/orig      -> origin/gh/bobrenjc93/549/orig
2025-09-07T07:36:03.7955972Z  * [new branch]              gh/bobrenjc93/550/base      -> origin/gh/bobrenjc93/550/base
2025-09-07T07:36:03.7957144Z  * [new branch]              gh/bobrenjc93/550/head      -> origin/gh/bobrenjc93/550/head
2025-09-07T07:36:03.7958357Z  * [new branch]              gh/bobrenjc93/550/orig      -> origin/gh/bobrenjc93/550/orig
2025-09-07T07:36:03.7960290Z  * [new branch]              gh/bobrenjc93/551/base      -> origin/gh/bobrenjc93/551/base
2025-09-07T07:36:03.7961634Z  * [new branch]              gh/bobrenjc93/551/head      -> origin/gh/bobrenjc93/551/head
2025-09-07T07:36:03.7962899Z  * [new branch]              gh/bobrenjc93/551/orig      -> origin/gh/bobrenjc93/551/orig
2025-09-07T07:36:03.7964896Z  * [new branch]              gh/bobrenjc93/552/base      -> origin/gh/bobrenjc93/552/base
2025-09-07T07:36:03.7965957Z  * [new branch]              gh/bobrenjc93/552/head      -> origin/gh/bobrenjc93/552/head
2025-09-07T07:36:03.7967375Z  * [new branch]              gh/bobrenjc93/552/orig      -> origin/gh/bobrenjc93/552/orig
2025-09-07T07:36:03.7968929Z  * [new branch]              gh/bobrenjc93/553/base      -> origin/gh/bobrenjc93/553/base
2025-09-07T07:36:03.7970185Z  * [new branch]              gh/bobrenjc93/553/head      -> origin/gh/bobrenjc93/553/head
2025-09-07T07:36:03.7971349Z  * [new branch]              gh/bobrenjc93/553/orig      -> origin/gh/bobrenjc93/553/orig
2025-09-07T07:36:03.7972991Z  * [new branch]              gh/bobrenjc93/554/base      -> origin/gh/bobrenjc93/554/base
2025-09-07T07:36:03.7974185Z  * [new branch]              gh/bobrenjc93/554/head      -> origin/gh/bobrenjc93/554/head
2025-09-07T07:36:03.7975367Z  * [new branch]              gh/bobrenjc93/554/orig      -> origin/gh/bobrenjc93/554/orig
2025-09-07T07:36:03.7977177Z  * [new branch]              gh/bobrenjc93/555/base      -> origin/gh/bobrenjc93/555/base
2025-09-07T07:36:03.7978406Z  * [new branch]              gh/bobrenjc93/555/head      -> origin/gh/bobrenjc93/555/head
2025-09-07T07:36:03.7979688Z  * [new branch]              gh/bobrenjc93/555/orig      -> origin/gh/bobrenjc93/555/orig
2025-09-07T07:36:03.7981443Z  * [new branch]              gh/bobrenjc93/556/base      -> origin/gh/bobrenjc93/556/base
2025-09-07T07:36:03.7982657Z  * [new branch]              gh/bobrenjc93/556/head      -> origin/gh/bobrenjc93/556/head
2025-09-07T07:36:03.7983875Z  * [new branch]              gh/bobrenjc93/556/orig      -> origin/gh/bobrenjc93/556/orig
2025-09-07T07:36:03.7986139Z  * [new branch]              gh/briancoutinho/2/base     -> origin/gh/briancoutinho/2/base
2025-09-07T07:36:03.7987482Z  * [new branch]              gh/briancoutinho/2/head     -> origin/gh/briancoutinho/2/head
2025-09-07T07:36:03.7989670Z  * [new branch]              gh/c00w/23/base             -> origin/gh/c00w/23/base
2025-09-07T07:36:03.7990853Z  * [new branch]              gh/c00w/23/head             -> origin/gh/c00w/23/head
2025-09-07T07:36:03.7992929Z  * [new branch]              gh/c00w/48/base             -> origin/gh/c00w/48/base
2025-09-07T07:36:03.7993820Z  * [new branch]              gh/c00w/48/head             -> origin/gh/c00w/48/head
2025-09-07T07:36:03.7995095Z  * [new branch]              gh/c00w/48/orig             -> origin/gh/c00w/48/orig
2025-09-07T07:36:03.7996935Z  * [new branch]              gh/c00w/53/base             -> origin/gh/c00w/53/base
2025-09-07T07:36:03.7998254Z  * [new branch]              gh/c00w/53/head             -> origin/gh/c00w/53/head
2025-09-07T07:36:03.7999459Z  * [new branch]              gh/c00w/53/orig             -> origin/gh/c00w/53/orig
2025-09-07T07:36:03.8001055Z  * [new branch]              gh/c00w/54/base             -> origin/gh/c00w/54/base
2025-09-07T07:36:03.8002293Z  * [new branch]              gh/c00w/54/head             -> origin/gh/c00w/54/head
2025-09-07T07:36:03.8003711Z  * [new branch]              gh/c00w/54/orig             -> origin/gh/c00w/54/orig
2025-09-07T07:36:03.8005545Z  * [new branch]              gh/c00w/55/base             -> origin/gh/c00w/55/base
2025-09-07T07:36:03.8006812Z  * [new branch]              gh/c00w/55/head             -> origin/gh/c00w/55/head
2025-09-07T07:36:03.8008188Z  * [new branch]              gh/c00w/55/orig             -> origin/gh/c00w/55/orig
2025-09-07T07:36:03.8009930Z  * [new branch]              gh/c00w/56/base             -> origin/gh/c00w/56/base
2025-09-07T07:36:03.8011357Z  * [new branch]              gh/c00w/56/head             -> origin/gh/c00w/56/head
2025-09-07T07:36:03.8012749Z  * [new branch]              gh/c00w/56/orig             -> origin/gh/c00w/56/orig
2025-09-07T07:36:03.8014879Z  * [new branch]              gh/clee2000/1/base          -> origin/gh/clee2000/1/base
2025-09-07T07:36:03.8016320Z  * [new branch]              gh/clee2000/1/head          -> origin/gh/clee2000/1/head
2025-09-07T07:36:03.8017374Z  * [new branch]              gh/clee2000/1/orig          -> origin/gh/clee2000/1/orig
2025-09-07T07:36:03.8019695Z  * [new branch]              gh/coconutruben/1/base      -> origin/gh/coconutruben/1/base
2025-09-07T07:36:03.8020976Z  * [new branch]              gh/coconutruben/1/head      -> origin/gh/coconutruben/1/head
2025-09-07T07:36:03.8022821Z  * [new branch]              gh/coconutruben/11/base     -> origin/gh/coconutruben/11/base
2025-09-07T07:36:03.8024150Z  * [new branch]              gh/coconutruben/11/head     -> origin/gh/coconutruben/11/head
2025-09-07T07:36:03.8025410Z  * [new branch]              gh/coconutruben/11/orig     -> origin/gh/coconutruben/11/orig
2025-09-07T07:36:03.8027508Z  * [new branch]              gh/coconutruben/12/base     -> origin/gh/coconutruben/12/base
2025-09-07T07:36:03.8028959Z  * [new branch]              gh/coconutruben/12/head     -> origin/gh/coconutruben/12/head
2025-09-07T07:36:03.8030383Z  * [new branch]              gh/coconutruben/12/orig     -> origin/gh/coconutruben/12/orig
2025-09-07T07:36:03.8032189Z  * [new branch]              gh/coconutruben/13/base     -> origin/gh/coconutruben/13/base
2025-09-07T07:36:03.8033480Z  * [new branch]              gh/coconutruben/13/head     -> origin/gh/coconutruben/13/head
2025-09-07T07:36:03.8034824Z  * [new branch]              gh/coconutruben/13/orig     -> origin/gh/coconutruben/13/orig
2025-09-07T07:36:03.8036740Z  * [new branch]              gh/coconutruben/14/base     -> origin/gh/coconutruben/14/base
2025-09-07T07:36:03.8038005Z  * [new branch]              gh/coconutruben/14/head     -> origin/gh/coconutruben/14/head
2025-09-07T07:36:03.8039201Z  * [new branch]              gh/coconutruben/14/orig     -> origin/gh/coconutruben/14/orig
2025-09-07T07:36:03.8041153Z  * [new branch]              gh/coconutruben/15/base     -> origin/gh/coconutruben/15/base
2025-09-07T07:36:03.8042498Z  * [new branch]              gh/coconutruben/15/head     -> origin/gh/coconutruben/15/head
2025-09-07T07:36:03.8043788Z  * [new branch]              gh/coconutruben/15/orig     -> origin/gh/coconutruben/15/orig
2025-09-07T07:36:03.8045454Z  * [new branch]              gh/coconutruben/16/base     -> origin/gh/coconutruben/16/base
2025-09-07T07:36:03.8046663Z  * [new branch]              gh/coconutruben/16/head     -> origin/gh/coconutruben/16/head
2025-09-07T07:36:03.8047842Z  * [new branch]              gh/coconutruben/16/orig     -> origin/gh/coconutruben/16/orig
2025-09-07T07:36:03.8049882Z  * [new branch]              gh/coconutruben/17/base     -> origin/gh/coconutruben/17/base
2025-09-07T07:36:03.8051195Z  * [new branch]              gh/coconutruben/17/head     -> origin/gh/coconutruben/17/head
2025-09-07T07:36:03.8052544Z  * [new branch]              gh/coconutruben/17/orig     -> origin/gh/coconutruben/17/orig
2025-09-07T07:36:03.8054313Z  * [new branch]              gh/coconutruben/18/base     -> origin/gh/coconutruben/18/base
2025-09-07T07:36:03.8055657Z  * [new branch]              gh/coconutruben/18/head     -> origin/gh/coconutruben/18/head
2025-09-07T07:36:03.8056826Z  * [new branch]              gh/coconutruben/18/orig     -> origin/gh/coconutruben/18/orig
2025-09-07T07:36:03.8058815Z  * [new branch]              gh/coconutruben/19/base     -> origin/gh/coconutruben/19/base
2025-09-07T07:36:03.8060127Z  * [new branch]              gh/coconutruben/19/head     -> origin/gh/coconutruben/19/head
2025-09-07T07:36:03.8061381Z  * [new branch]              gh/coconutruben/19/orig     -> origin/gh/coconutruben/19/orig
2025-09-07T07:36:03.8063270Z  * [new branch]              gh/coconutruben/20/base     -> origin/gh/coconutruben/20/base
2025-09-07T07:36:03.8064513Z  * [new branch]              gh/coconutruben/20/head     -> origin/gh/coconutruben/20/head
2025-09-07T07:36:03.8065823Z  * [new branch]              gh/coconutruben/20/orig     -> origin/gh/coconutruben/20/orig
2025-09-07T07:36:03.8067703Z  * [new branch]              gh/coconutruben/21/base     -> origin/gh/coconutruben/21/base
2025-09-07T07:36:03.8068969Z  * [new branch]              gh/coconutruben/21/head     -> origin/gh/coconutruben/21/head
2025-09-07T07:36:03.8070089Z  * [new branch]              gh/coconutruben/21/orig     -> origin/gh/coconutruben/21/orig
2025-09-07T07:36:03.8071860Z  * [new branch]              gh/coconutruben/22/base     -> origin/gh/coconutruben/22/base
2025-09-07T07:36:03.8073161Z  * [new branch]              gh/coconutruben/22/head     -> origin/gh/coconutruben/22/head
2025-09-07T07:36:03.8074512Z  * [new branch]              gh/coconutruben/22/orig     -> origin/gh/coconutruben/22/orig
2025-09-07T07:36:03.8076312Z  * [new branch]              gh/coconutruben/24/base     -> origin/gh/coconutruben/24/base
2025-09-07T07:36:03.8077595Z  * [new branch]              gh/coconutruben/24/head     -> origin/gh/coconutruben/24/head
2025-09-07T07:36:03.8078839Z  * [new branch]              gh/coconutruben/24/orig     -> origin/gh/coconutruben/24/orig
2025-09-07T07:36:03.8080954Z  * [new branch]              gh/coconutruben/25/base     -> origin/gh/coconutruben/25/base
2025-09-07T07:36:03.8082432Z  * [new branch]              gh/coconutruben/25/head     -> origin/gh/coconutruben/25/head
2025-09-07T07:36:03.8083837Z  * [new branch]              gh/coconutruben/25/orig     -> origin/gh/coconutruben/25/orig
2025-09-07T07:36:03.8085721Z  * [new branch]              gh/coconutruben/28/base     -> origin/gh/coconutruben/28/base
2025-09-07T07:36:03.8086968Z  * [new branch]              gh/coconutruben/28/head     -> origin/gh/coconutruben/28/head
2025-09-07T07:36:03.8088235Z  * [new branch]              gh/coconutruben/28/orig     -> origin/gh/coconutruben/28/orig
2025-09-07T07:36:03.8090086Z  * [new branch]              gh/coconutruben/29/base     -> origin/gh/coconutruben/29/base
2025-09-07T07:36:03.8091338Z  * [new branch]              gh/coconutruben/29/head     -> origin/gh/coconutruben/29/head
2025-09-07T07:36:03.8092562Z  * [new branch]              gh/coconutruben/29/orig     -> origin/gh/coconutruben/29/orig
2025-09-07T07:36:03.8094392Z  * [new branch]              gh/coconutruben/30/base     -> origin/gh/coconutruben/30/base
2025-09-07T07:36:03.8095647Z  * [new branch]              gh/coconutruben/30/head     -> origin/gh/coconutruben/30/head
2025-09-07T07:36:03.8096964Z  * [new branch]              gh/coconutruben/30/orig     -> origin/gh/coconutruben/30/orig
2025-09-07T07:36:03.8099304Z  * [new branch]              gh/coconutruben/31/base     -> origin/gh/coconutruben/31/base
2025-09-07T07:36:03.8100647Z  * [new branch]              gh/coconutruben/31/head     -> origin/gh/coconutruben/31/head
2025-09-07T07:36:03.8101926Z  * [new branch]              gh/coconutruben/31/orig     -> origin/gh/coconutruben/31/orig
2025-09-07T07:36:03.8108113Z  * [new branch]              gh/coconutruben/32/base     -> origin/gh/coconutruben/32/base
2025-09-07T07:36:03.8109471Z  * [new branch]              gh/coconutruben/32/head     -> origin/gh/coconutruben/32/head
2025-09-07T07:36:03.8110704Z  * [new branch]              gh/coconutruben/32/orig     -> origin/gh/coconutruben/32/orig
2025-09-07T07:36:03.8112712Z  * [new branch]              gh/coconutruben/33/base     -> origin/gh/coconutruben/33/base
2025-09-07T07:36:03.8114113Z  * [new branch]              gh/coconutruben/33/head     -> origin/gh/coconutruben/33/head
2025-09-07T07:36:03.8115465Z  * [new branch]              gh/coconutruben/33/orig     -> origin/gh/coconutruben/33/orig
2025-09-07T07:36:03.8117553Z  * [new branch]              gh/coconutruben/34/base     -> origin/gh/coconutruben/34/base
2025-09-07T07:36:03.8118768Z  * [new branch]              gh/coconutruben/34/head     -> origin/gh/coconutruben/34/head
2025-09-07T07:36:03.8119914Z  * [new branch]              gh/coconutruben/34/orig     -> origin/gh/coconutruben/34/orig
2025-09-07T07:36:03.8121717Z  * [new branch]              gh/coconutruben/35/base     -> origin/gh/coconutruben/35/base
2025-09-07T07:36:03.8122983Z  * [new branch]              gh/coconutruben/35/head     -> origin/gh/coconutruben/35/head
2025-09-07T07:36:03.8124409Z  * [new branch]              gh/coconutruben/35/orig     -> origin/gh/coconutruben/35/orig
2025-09-07T07:36:03.8127223Z  * [new branch]              gh/coconutruben/36/base     -> origin/gh/coconutruben/36/base
2025-09-07T07:36:03.8128832Z  * [new branch]              gh/coconutruben/36/head     -> origin/gh/coconutruben/36/head
2025-09-07T07:36:03.8130620Z  * [new branch]              gh/coconutruben/36/orig     -> origin/gh/coconutruben/36/orig
2025-09-07T07:36:03.8132622Z  * [new branch]              gh/coconutruben/37/base     -> origin/gh/coconutruben/37/base
2025-09-07T07:36:03.8133871Z  * [new branch]              gh/coconutruben/37/head     -> origin/gh/coconutruben/37/head
2025-09-07T07:36:03.8135113Z  * [new branch]              gh/coconutruben/37/orig     -> origin/gh/coconutruben/37/orig
2025-09-07T07:36:03.8137006Z  * [new branch]              gh/coconutruben/38/base     -> origin/gh/coconutruben/38/base
2025-09-07T07:36:03.8138458Z  * [new branch]              gh/coconutruben/38/head     -> origin/gh/coconutruben/38/head
2025-09-07T07:36:03.8139694Z  * [new branch]              gh/coconutruben/38/orig     -> origin/gh/coconutruben/38/orig
2025-09-07T07:36:03.8141624Z  * [new branch]              gh/coconutruben/39/base     -> origin/gh/coconutruben/39/base
2025-09-07T07:36:03.8142780Z  * [new branch]              gh/coconutruben/39/head     -> origin/gh/coconutruben/39/head
2025-09-07T07:36:03.8144001Z  * [new branch]              gh/coconutruben/39/orig     -> origin/gh/coconutruben/39/orig
2025-09-07T07:36:03.8146077Z  * [new branch]              gh/coconutruben/40/base     -> origin/gh/coconutruben/40/base
2025-09-07T07:36:03.8147235Z  * [new branch]              gh/coconutruben/40/head     -> origin/gh/coconutruben/40/head
2025-09-07T07:36:03.8148456Z  * [new branch]              gh/coconutruben/40/orig     -> origin/gh/coconutruben/40/orig
2025-09-07T07:36:03.8150523Z  * [new branch]              gh/coconutruben/41/base     -> origin/gh/coconutruben/41/base
2025-09-07T07:36:03.8151776Z  * [new branch]              gh/coconutruben/41/head     -> origin/gh/coconutruben/41/head
2025-09-07T07:36:03.8153031Z  * [new branch]              gh/coconutruben/41/orig     -> origin/gh/coconutruben/41/orig
2025-09-07T07:36:03.8154965Z  * [new branch]              gh/coconutruben/42/base     -> origin/gh/coconutruben/42/base
2025-09-07T07:36:03.8156342Z  * [new branch]              gh/coconutruben/42/head     -> origin/gh/coconutruben/42/head
2025-09-07T07:36:03.8157597Z  * [new branch]              gh/coconutruben/42/orig     -> origin/gh/coconutruben/42/orig
2025-09-07T07:36:03.8159577Z  * [new branch]              gh/coconutruben/43/base     -> origin/gh/coconutruben/43/base
2025-09-07T07:36:03.8160834Z  * [new branch]              gh/coconutruben/43/head     -> origin/gh/coconutruben/43/head
2025-09-07T07:36:03.8162055Z  * [new branch]              gh/coconutruben/43/orig     -> origin/gh/coconutruben/43/orig
2025-09-07T07:36:03.8164103Z  * [new branch]              gh/coconutruben/44/base     -> origin/gh/coconutruben/44/base
2025-09-07T07:36:03.8165385Z  * [new branch]              gh/coconutruben/44/head     -> origin/gh/coconutruben/44/head
2025-09-07T07:36:03.8166668Z  * [new branch]              gh/coconutruben/44/orig     -> origin/gh/coconutruben/44/orig
2025-09-07T07:36:03.8168575Z  * [new branch]              gh/coconutruben/45/base     -> origin/gh/coconutruben/45/base
2025-09-07T07:36:03.8169830Z  * [new branch]              gh/coconutruben/45/head     -> origin/gh/coconutruben/45/head
2025-09-07T07:36:03.8171119Z  * [new branch]              gh/coconutruben/45/orig     -> origin/gh/coconutruben/45/orig
2025-09-07T07:36:03.8172913Z  * [new branch]              gh/coconutruben/46/base     -> origin/gh/coconutruben/46/base
2025-09-07T07:36:03.8174201Z  * [new branch]              gh/coconutruben/46/head     -> origin/gh/coconutruben/46/head
2025-09-07T07:36:03.8175599Z  * [new branch]              gh/coconutruben/46/orig     -> origin/gh/coconutruben/46/orig
2025-09-07T07:36:03.8177463Z  * [new branch]              gh/coconutruben/47/base     -> origin/gh/coconutruben/47/base
2025-09-07T07:36:03.8178763Z  * [new branch]              gh/coconutruben/47/head     -> origin/gh/coconutruben/47/head
2025-09-07T07:36:03.8180027Z  * [new branch]              gh/coconutruben/47/orig     -> origin/gh/coconutruben/47/orig
2025-09-07T07:36:03.8182153Z  * [new branch]              gh/coconutruben/48/base     -> origin/gh/coconutruben/48/base
2025-09-07T07:36:03.8183480Z  * [new branch]              gh/coconutruben/48/head     -> origin/gh/coconutruben/48/head
2025-09-07T07:36:03.8184712Z  * [new branch]              gh/coconutruben/48/orig     -> origin/gh/coconutruben/48/orig
2025-09-07T07:36:03.8186742Z  * [new branch]              gh/coconutruben/49/base     -> origin/gh/coconutruben/49/base
2025-09-07T07:36:03.8187997Z  * [new branch]              gh/coconutruben/49/head     -> origin/gh/coconutruben/49/head
2025-09-07T07:36:03.8189220Z  * [new branch]              gh/coconutruben/49/orig     -> origin/gh/coconutruben/49/orig
2025-09-07T07:36:03.8191078Z  * [new branch]              gh/coconutruben/50/base     -> origin/gh/coconutruben/50/base
2025-09-07T07:36:03.8192400Z  * [new branch]              gh/coconutruben/50/head     -> origin/gh/coconutruben/50/head
2025-09-07T07:36:03.8193773Z  * [new branch]              gh/coconutruben/50/orig     -> origin/gh/coconutruben/50/orig
2025-09-07T07:36:03.8195548Z  * [new branch]              gh/coconutruben/51/base     -> origin/gh/coconutruben/51/base
2025-09-07T07:36:03.8196844Z  * [new branch]              gh/coconutruben/51/head     -> origin/gh/coconutruben/51/head
2025-09-07T07:36:03.8198084Z  * [new branch]              gh/coconutruben/51/orig     -> origin/gh/coconutruben/51/orig
2025-09-07T07:36:03.8200130Z  * [new branch]              gh/coconutruben/52/base     -> origin/gh/coconutruben/52/base
2025-09-07T07:36:03.8201409Z  * [new branch]              gh/coconutruben/52/head     -> origin/gh/coconutruben/52/head
2025-09-07T07:36:03.8202705Z  * [new branch]              gh/coconutruben/52/orig     -> origin/gh/coconutruben/52/orig
2025-09-07T07:36:03.8204762Z  * [new branch]              gh/coconutruben/53/base     -> origin/gh/coconutruben/53/base
2025-09-07T07:36:03.8205941Z  * [new branch]              gh/coconutruben/53/head     -> origin/gh/coconutruben/53/head
2025-09-07T07:36:03.8207171Z  * [new branch]              gh/coconutruben/53/orig     -> origin/gh/coconutruben/53/orig
2025-09-07T07:36:03.8209016Z  * [new branch]              gh/coconutruben/54/base     -> origin/gh/coconutruben/54/base
2025-09-07T07:36:03.8210295Z  * [new branch]              gh/coconutruben/54/head     -> origin/gh/coconutruben/54/head
2025-09-07T07:36:03.8211519Z  * [new branch]              gh/coconutruben/54/orig     -> origin/gh/coconutruben/54/orig
2025-09-07T07:36:03.8213398Z  * [new branch]              gh/coconutruben/55/base     -> origin/gh/coconutruben/55/base
2025-09-07T07:36:03.8214592Z  * [new branch]              gh/coconutruben/55/head     -> origin/gh/coconutruben/55/head
2025-09-07T07:36:03.8215885Z  * [new branch]              gh/coconutruben/55/orig     -> origin/gh/coconutruben/55/orig
2025-09-07T07:36:03.8218033Z  * [new branch]              gh/coconutruben/56/base     -> origin/gh/coconutruben/56/base
2025-09-07T07:36:03.8219288Z  * [new branch]              gh/coconutruben/56/head     -> origin/gh/coconutruben/56/head
2025-09-07T07:36:03.8220514Z  * [new branch]              gh/coconutruben/56/orig     -> origin/gh/coconutruben/56/orig
2025-09-07T07:36:03.8222398Z  * [new branch]              gh/coconutruben/57/base     -> origin/gh/coconutruben/57/base
2025-09-07T07:36:03.8223693Z  * [new branch]              gh/coconutruben/57/head     -> origin/gh/coconutruben/57/head
2025-09-07T07:36:03.8224960Z  * [new branch]              gh/coconutruben/57/orig     -> origin/gh/coconutruben/57/orig
2025-09-07T07:36:03.8227026Z  * [new branch]              gh/coconutruben/58/base     -> origin/gh/coconutruben/58/base
2025-09-07T07:36:03.8228563Z  * [new branch]              gh/coconutruben/58/head     -> origin/gh/coconutruben/58/head
2025-09-07T07:36:03.8229675Z  * [new branch]              gh/coconutruben/58/orig     -> origin/gh/coconutruben/58/orig
2025-09-07T07:36:03.8231722Z  * [new branch]              gh/coconutruben/59/base     -> origin/gh/coconutruben/59/base
2025-09-07T07:36:03.8232865Z  * [new branch]              gh/coconutruben/59/head     -> origin/gh/coconutruben/59/head
2025-09-07T07:36:03.8234036Z  * [new branch]              gh/coconutruben/59/orig     -> origin/gh/coconutruben/59/orig
2025-09-07T07:36:03.8235921Z  * [new branch]              gh/coconutruben/60/base     -> origin/gh/coconutruben/60/base
2025-09-07T07:36:03.8237173Z  * [new branch]              gh/coconutruben/60/head     -> origin/gh/coconutruben/60/head
2025-09-07T07:36:03.8238474Z  * [new branch]              gh/coconutruben/60/orig     -> origin/gh/coconutruben/60/orig
2025-09-07T07:36:03.8240233Z  * [new branch]              gh/coconutruben/61/base     -> origin/gh/coconutruben/61/base
2025-09-07T07:36:03.8241571Z  * [new branch]              gh/coconutruben/61/head     -> origin/gh/coconutruben/61/head
2025-09-07T07:36:03.8242767Z  * [new branch]              gh/coconutruben/61/orig     -> origin/gh/coconutruben/61/orig
2025-09-07T07:36:03.8244690Z  * [new branch]              gh/coconutruben/62/base     -> origin/gh/coconutruben/62/base
2025-09-07T07:36:03.8245929Z  * [new branch]              gh/coconutruben/62/head     -> origin/gh/coconutruben/62/head
2025-09-07T07:36:03.8247218Z  * [new branch]              gh/coconutruben/62/orig     -> origin/gh/coconutruben/62/orig
2025-09-07T07:36:03.8249166Z  * [new branch]              gh/coconutruben/63/base     -> origin/gh/coconutruben/63/base
2025-09-07T07:36:03.8250424Z  * [new branch]              gh/coconutruben/63/head     -> origin/gh/coconutruben/63/head
2025-09-07T07:36:03.8251655Z  * [new branch]              gh/coconutruben/63/orig     -> origin/gh/coconutruben/63/orig
2025-09-07T07:36:03.8253564Z  * [new branch]              gh/coconutruben/64/base     -> origin/gh/coconutruben/64/base
2025-09-07T07:36:03.8254837Z  * [new branch]              gh/coconutruben/64/head     -> origin/gh/coconutruben/64/head
2025-09-07T07:36:03.8256056Z  * [new branch]              gh/coconutruben/64/orig     -> origin/gh/coconutruben/64/orig
2025-09-07T07:36:03.8257976Z  * [new branch]              gh/coconutruben/65/base     -> origin/gh/coconutruben/65/base
2025-09-07T07:36:03.8259454Z  * [new branch]              gh/coconutruben/65/head     -> origin/gh/coconutruben/65/head
2025-09-07T07:36:03.8260469Z  * [new branch]              gh/coconutruben/65/orig     -> origin/gh/coconutruben/65/orig
2025-09-07T07:36:03.8262283Z  * [new branch]              gh/coconutruben/66/base     -> origin/gh/coconutruben/66/base
2025-09-07T07:36:03.8263491Z  * [new branch]              gh/coconutruben/66/head     -> origin/gh/coconutruben/66/head
2025-09-07T07:36:03.8264668Z  * [new branch]              gh/coconutruben/66/orig     -> origin/gh/coconutruben/66/orig
2025-09-07T07:36:03.8267170Z  * [new branch]              gh/codingwithsurya/12/base  -> origin/gh/codingwithsurya/12/base
2025-09-07T07:36:03.8268688Z  * [new branch]              gh/codingwithsurya/12/head  -> origin/gh/codingwithsurya/12/head
2025-09-07T07:36:03.8270115Z  * [new branch]              gh/codingwithsurya/12/orig  -> origin/gh/codingwithsurya/12/orig
2025-09-07T07:36:03.8271798Z  * [new branch]              gh/codingwithsurya/14/base  -> origin/gh/codingwithsurya/14/base
2025-09-07T07:36:03.8273028Z  * [new branch]              gh/codingwithsurya/14/head  -> origin/gh/codingwithsurya/14/head
2025-09-07T07:36:03.8274301Z  * [new branch]              gh/codingwithsurya/14/orig  -> origin/gh/codingwithsurya/14/orig
2025-09-07T07:36:03.8276201Z  * [new branch]              gh/codingwithsurya/15/base  -> origin/gh/codingwithsurya/15/base
2025-09-07T07:36:03.8277471Z  * [new branch]              gh/codingwithsurya/15/head  -> origin/gh/codingwithsurya/15/head
2025-09-07T07:36:03.8278733Z  * [new branch]              gh/codingwithsurya/15/orig  -> origin/gh/codingwithsurya/15/orig
2025-09-07T07:36:03.8280598Z  * [new branch]              gh/codingwithsurya/16/base  -> origin/gh/codingwithsurya/16/base
2025-09-07T07:36:03.8281810Z  * [new branch]              gh/codingwithsurya/16/head  -> origin/gh/codingwithsurya/16/head
2025-09-07T07:36:03.8283055Z  * [new branch]              gh/codingwithsurya/16/orig  -> origin/gh/codingwithsurya/16/orig
2025-09-07T07:36:03.8284958Z  * [new branch]              gh/codingwithsurya/17/base  -> origin/gh/codingwithsurya/17/base
2025-09-07T07:36:03.8286261Z  * [new branch]              gh/codingwithsurya/17/head  -> origin/gh/codingwithsurya/17/head
2025-09-07T07:36:03.8287524Z  * [new branch]              gh/codingwithsurya/17/orig  -> origin/gh/codingwithsurya/17/orig
2025-09-07T07:36:03.8289330Z  * [new branch]              gh/codingwithsurya/18/base  -> origin/gh/codingwithsurya/18/base
2025-09-07T07:36:03.8290603Z  * [new branch]              gh/codingwithsurya/18/head  -> origin/gh/codingwithsurya/18/head
2025-09-07T07:36:03.8291813Z  * [new branch]              gh/codingwithsurya/18/orig  -> origin/gh/codingwithsurya/18/orig
2025-09-07T07:36:03.8293722Z  * [new branch]              gh/codingwithsurya/19/base  -> origin/gh/codingwithsurya/19/base
2025-09-07T07:36:03.8294924Z  * [new branch]              gh/codingwithsurya/19/head  -> origin/gh/codingwithsurya/19/head
2025-09-07T07:36:03.8296180Z  * [new branch]              gh/codingwithsurya/19/orig  -> origin/gh/codingwithsurya/19/orig
2025-09-07T07:36:03.8298049Z  * [new branch]              gh/codingwithsurya/20/base  -> origin/gh/codingwithsurya/20/base
2025-09-07T07:36:03.8299266Z  * [new branch]              gh/codingwithsurya/20/head  -> origin/gh/codingwithsurya/20/head
2025-09-07T07:36:03.8300460Z  * [new branch]              gh/codingwithsurya/20/orig  -> origin/gh/codingwithsurya/20/orig
2025-09-07T07:36:03.8302319Z  * [new branch]              gh/codingwithsurya/21/base  -> origin/gh/codingwithsurya/21/base
2025-09-07T07:36:03.8303855Z  * [new branch]              gh/codingwithsurya/21/head  -> origin/gh/codingwithsurya/21/head
2025-09-07T07:36:03.8305374Z  * [new branch]              gh/codingwithsurya/21/orig  -> origin/gh/codingwithsurya/21/orig
2025-09-07T07:36:03.8307578Z  * [new branch]              gh/colinchan15/1/base       -> origin/gh/colinchan15/1/base
2025-09-07T07:36:03.8308776Z  * [new branch]              gh/colinchan15/1/head       -> origin/gh/colinchan15/1/head
2025-09-07T07:36:03.8310355Z  * [new branch]              gh/colinchan15/2/base       -> origin/gh/colinchan15/2/base
2025-09-07T07:36:03.8311482Z  * [new branch]              gh/colinchan15/2/head       -> origin/gh/colinchan15/2/head
2025-09-07T07:36:03.8313104Z  * [new branch]              gh/colinchan15/3/base       -> origin/gh/colinchan15/3/base
2025-09-07T07:36:03.8314182Z  * [new branch]              gh/colinchan15/3/head       -> origin/gh/colinchan15/3/head
2025-09-07T07:36:03.8315815Z  * [new branch]              gh/colinchan15/6/base       -> origin/gh/colinchan15/6/base
2025-09-07T07:36:03.8317005Z  * [new branch]              gh/colinchan15/6/head       -> origin/gh/colinchan15/6/head
2025-09-07T07:36:03.8319321Z  * [new branch]              gh/davidberard98/382/base   -> origin/gh/davidberard98/382/base
2025-09-07T07:36:03.8320734Z  * [new branch]              gh/davidberard98/382/head   -> origin/gh/davidberard98/382/head
2025-09-07T07:36:03.8321936Z  * [new branch]              gh/davidberard98/382/orig   -> origin/gh/davidberard98/382/orig
2025-09-07T07:36:03.8323663Z  * [new branch]              gh/davidberard98/386/base   -> origin/gh/davidberard98/386/base
2025-09-07T07:36:03.8324892Z  * [new branch]              gh/davidberard98/386/head   -> origin/gh/davidberard98/386/head
2025-09-07T07:36:03.8326175Z  * [new branch]              gh/davidberard98/386/orig   -> origin/gh/davidberard98/386/orig
2025-09-07T07:36:03.8327936Z  * [new branch]              gh/davidberard98/391/base   -> origin/gh/davidberard98/391/base
2025-09-07T07:36:03.8329141Z  * [new branch]              gh/davidberard98/391/head   -> origin/gh/davidberard98/391/head
2025-09-07T07:36:03.8330238Z  * [new branch]              gh/davidberard98/391/orig   -> origin/gh/davidberard98/391/orig
2025-09-07T07:36:03.8332335Z  * [new branch]              gh/davidberard98/392/base   -> origin/gh/davidberard98/392/base
2025-09-07T07:36:03.8333132Z  * [new branch]              gh/davidberard98/392/head   -> origin/gh/davidberard98/392/head
2025-09-07T07:36:03.8334328Z  * [new branch]              gh/davidberard98/392/orig   -> origin/gh/davidberard98/392/orig
2025-09-07T07:36:03.8336366Z  * [new branch]              gh/davidberard98/394/base   -> origin/gh/davidberard98/394/base
2025-09-07T07:36:03.8337499Z  * [new branch]              gh/davidberard98/394/head   -> origin/gh/davidberard98/394/head
2025-09-07T07:36:03.8338814Z  * [new branch]              gh/davidberard98/394/orig   -> origin/gh/davidberard98/394/orig
2025-09-07T07:36:03.8340541Z  * [new branch]              gh/davidberard98/396/base   -> origin/gh/davidberard98/396/base
2025-09-07T07:36:03.8341714Z  * [new branch]              gh/davidberard98/396/head   -> origin/gh/davidberard98/396/head
2025-09-07T07:36:03.8343129Z  * [new branch]              gh/davidberard98/396/orig   -> origin/gh/davidberard98/396/orig
2025-09-07T07:36:03.8345022Z  * [new branch]              gh/davidberard98/397/base   -> origin/gh/davidberard98/397/base
2025-09-07T07:36:03.8346226Z  * [new branch]              gh/davidberard98/397/head   -> origin/gh/davidberard98/397/head
2025-09-07T07:36:03.8347495Z  * [new branch]              gh/davidberard98/397/orig   -> origin/gh/davidberard98/397/orig
2025-09-07T07:36:03.8349231Z  * [new branch]              gh/davidberard98/398/base   -> origin/gh/davidberard98/398/base
2025-09-07T07:36:03.8350397Z  * [new branch]              gh/davidberard98/398/head   -> origin/gh/davidberard98/398/head
2025-09-07T07:36:03.8351647Z  * [new branch]              gh/davidberard98/398/orig   -> origin/gh/davidberard98/398/orig
2025-09-07T07:36:03.8353505Z  * [new branch]              gh/davidberard98/399/base   -> origin/gh/davidberard98/399/base
2025-09-07T07:36:03.8354739Z  * [new branch]              gh/davidberard98/399/head   -> origin/gh/davidberard98/399/head
2025-09-07T07:36:03.8355946Z  * [new branch]              gh/davidberard98/399/orig   -> origin/gh/davidberard98/399/orig
2025-09-07T07:36:03.8357687Z  * [new branch]              gh/davidberard98/400/base   -> origin/gh/davidberard98/400/base
2025-09-07T07:36:03.8358955Z  * [new branch]              gh/davidberard98/400/head   -> origin/gh/davidberard98/400/head
2025-09-07T07:36:03.8360162Z  * [new branch]              gh/davidberard98/400/orig   -> origin/gh/davidberard98/400/orig
2025-09-07T07:36:03.8361843Z  * [new branch]              gh/davidberard98/401/base   -> origin/gh/davidberard98/401/base
2025-09-07T07:36:03.8363076Z  * [new branch]              gh/davidberard98/401/head   -> origin/gh/davidberard98/401/head
2025-09-07T07:36:03.8364337Z  * [new branch]              gh/davidberard98/401/orig   -> origin/gh/davidberard98/401/orig
2025-09-07T07:36:03.8366045Z  * [new branch]              gh/davidberard98/402/base   -> origin/gh/davidberard98/402/base
2025-09-07T07:36:03.8367249Z  * [new branch]              gh/davidberard98/402/head   -> origin/gh/davidberard98/402/head
2025-09-07T07:36:03.8368445Z  * [new branch]              gh/davidberard98/402/orig   -> origin/gh/davidberard98/402/orig
2025-09-07T07:36:03.8370235Z  * [new branch]              gh/davidberard98/403/base   -> origin/gh/davidberard98/403/base
2025-09-07T07:36:03.8371434Z  * [new branch]              gh/davidberard98/403/head   -> origin/gh/davidberard98/403/head
2025-09-07T07:36:03.8372624Z  * [new branch]              gh/davidberard98/403/orig   -> origin/gh/davidberard98/403/orig
2025-09-07T07:36:03.8374490Z  * [new branch]              gh/davidberard98/404/base   -> origin/gh/davidberard98/404/base
2025-09-07T07:36:03.8375688Z  * [new branch]              gh/davidberard98/404/head   -> origin/gh/davidberard98/404/head
2025-09-07T07:36:03.8376784Z  * [new branch]              gh/davidberard98/404/orig   -> origin/gh/davidberard98/404/orig
2025-09-07T07:36:03.8378700Z  * [new branch]              gh/davidberard98/405/base   -> origin/gh/davidberard98/405/base
2025-09-07T07:36:03.8379921Z  * [new branch]              gh/davidberard98/405/head   -> origin/gh/davidberard98/405/head
2025-09-07T07:36:03.8381097Z  * [new branch]              gh/davidberard98/405/orig   -> origin/gh/davidberard98/405/orig
2025-09-07T07:36:03.8382940Z  * [new branch]              gh/davidberard98/406/base   -> origin/gh/davidberard98/406/base
2025-09-07T07:36:03.8384226Z  * [new branch]              gh/davidberard98/406/head   -> origin/gh/davidberard98/406/head
2025-09-07T07:36:03.8385500Z  * [new branch]              gh/davidberard98/406/orig   -> origin/gh/davidberard98/406/orig
2025-09-07T07:36:03.8387931Z  * [new branch]              gh/davidberard98/407/base   -> origin/gh/davidberard98/407/base
2025-09-07T07:36:03.8389126Z  * [new branch]              gh/davidberard98/407/head   -> origin/gh/davidberard98/407/head
2025-09-07T07:36:03.8390375Z  * [new branch]              gh/davidberard98/407/orig   -> origin/gh/davidberard98/407/orig
2025-09-07T07:36:03.8392161Z  * [new branch]              gh/davidberard98/408/base   -> origin/gh/davidberard98/408/base
2025-09-07T07:36:03.8393332Z  * [new branch]              gh/davidberard98/408/head   -> origin/gh/davidberard98/408/head
2025-09-07T07:36:03.8394494Z  * [new branch]              gh/davidberard98/408/orig   -> origin/gh/davidberard98/408/orig
2025-09-07T07:36:03.8396163Z  * [new branch]              gh/davidberard98/409/base   -> origin/gh/davidberard98/409/base
2025-09-07T07:36:03.8397423Z  * [new branch]              gh/davidberard98/409/head   -> origin/gh/davidberard98/409/head
2025-09-07T07:36:03.8398712Z  * [new branch]              gh/davidberard98/409/orig   -> origin/gh/davidberard98/409/orig
2025-09-07T07:36:03.8400888Z  * [new branch]              gh/desertfire/594/base      -> origin/gh/desertfire/594/base
2025-09-07T07:36:03.8402078Z  * [new branch]              gh/desertfire/594/head      -> origin/gh/desertfire/594/head
2025-09-07T07:36:03.8403439Z  * [new branch]              gh/desertfire/594/orig      -> origin/gh/desertfire/594/orig
2025-09-07T07:36:03.8405308Z  * [new branch]              gh/desertfire/595/base      -> origin/gh/desertfire/595/base
2025-09-07T07:36:03.8406443Z  * [new branch]              gh/desertfire/595/head      -> origin/gh/desertfire/595/head
2025-09-07T07:36:03.8407656Z  * [new branch]              gh/desertfire/595/orig      -> origin/gh/desertfire/595/orig
2025-09-07T07:36:03.8409467Z  * [new branch]              gh/desertfire/597/base      -> origin/gh/desertfire/597/base
2025-09-07T07:36:03.8410646Z  * [new branch]              gh/desertfire/597/head      -> origin/gh/desertfire/597/head
2025-09-07T07:36:03.8411888Z  * [new branch]              gh/desertfire/597/orig      -> origin/gh/desertfire/597/orig
2025-09-07T07:36:03.8414089Z  * [new branch]              gh/dharakk/1/base           -> origin/gh/dharakk/1/base
2025-09-07T07:36:03.8415443Z  * [new branch]              gh/dharakk/1/head           -> origin/gh/dharakk/1/head
2025-09-07T07:36:03.8417633Z  * [new branch]              gh/drisspg/149/base         -> origin/gh/drisspg/149/base
2025-09-07T07:36:03.8418866Z  * [new branch]              gh/drisspg/149/head         -> origin/gh/drisspg/149/head
2025-09-07T07:36:03.8420144Z  * [new branch]              gh/drisspg/149/orig         -> origin/gh/drisspg/149/orig
2025-09-07T07:36:03.8421833Z  * [new branch]              gh/drisspg/159/base         -> origin/gh/drisspg/159/base
2025-09-07T07:36:03.8422999Z  * [new branch]              gh/drisspg/159/head         -> origin/gh/drisspg/159/head
2025-09-07T07:36:03.8424187Z  * [new branch]              gh/drisspg/159/orig         -> origin/gh/drisspg/159/orig
2025-09-07T07:36:03.8426011Z  * [new branch]              gh/drisspg/166/base         -> origin/gh/drisspg/166/base
2025-09-07T07:36:03.8427111Z  * [new branch]              gh/drisspg/166/head         -> origin/gh/drisspg/166/head
2025-09-07T07:36:03.8428320Z  * [new branch]              gh/drisspg/166/orig         -> origin/gh/drisspg/166/orig
2025-09-07T07:36:03.8430019Z  * [new branch]              gh/drisspg/170/base         -> origin/gh/drisspg/170/base
2025-09-07T07:36:03.8431189Z  * [new branch]              gh/drisspg/170/head         -> origin/gh/drisspg/170/head
2025-09-07T07:36:03.8432413Z  * [new branch]              gh/drisspg/170/orig         -> origin/gh/drisspg/170/orig
2025-09-07T07:36:03.8434105Z  * [new branch]              gh/drisspg/173/base         -> origin/gh/drisspg/173/base
2025-09-07T07:36:03.8435291Z  * [new branch]              gh/drisspg/173/head         -> origin/gh/drisspg/173/head
2025-09-07T07:36:03.8436565Z  * [new branch]              gh/drisspg/173/orig         -> origin/gh/drisspg/173/orig
2025-09-07T07:36:03.8438266Z  * [new branch]              gh/drisspg/177/base         -> origin/gh/drisspg/177/base
2025-09-07T07:36:03.8439463Z  * [new branch]              gh/drisspg/177/head         -> origin/gh/drisspg/177/head
2025-09-07T07:36:03.8440625Z  * [new branch]              gh/drisspg/177/orig         -> origin/gh/drisspg/177/orig
2025-09-07T07:36:03.8442316Z  * [new branch]              gh/drisspg/178/base         -> origin/gh/drisspg/178/base
2025-09-07T07:36:03.8443497Z  * [new branch]              gh/drisspg/178/head         -> origin/gh/drisspg/178/head
2025-09-07T07:36:03.8444586Z  * [new branch]              gh/drisspg/178/orig         -> origin/gh/drisspg/178/orig
2025-09-07T07:36:03.8446302Z  * [new branch]              gh/drisspg/180/base         -> origin/gh/drisspg/180/base
2025-09-07T07:36:03.8447580Z  * [new branch]              gh/drisspg/180/head         -> origin/gh/drisspg/180/head
2025-09-07T07:36:03.8448740Z  * [new branch]              gh/drisspg/180/orig         -> origin/gh/drisspg/180/orig
2025-09-07T07:36:03.8450532Z  * [new branch]              gh/drisspg/181/base         -> origin/gh/drisspg/181/base
2025-09-07T07:36:03.8451773Z  * [new branch]              gh/drisspg/181/head         -> origin/gh/drisspg/181/head
2025-09-07T07:36:03.8453113Z  * [new branch]              gh/drisspg/181/orig         -> origin/gh/drisspg/181/orig
2025-09-07T07:36:03.8454822Z  * [new branch]              gh/drisspg/182/base         -> origin/gh/drisspg/182/base
2025-09-07T07:36:03.8456039Z  * [new branch]              gh/drisspg/182/head         -> origin/gh/drisspg/182/head
2025-09-07T07:36:03.8457723Z  * [new branch]              gh/drisspg/183/base         -> origin/gh/drisspg/183/base
2025-09-07T07:36:03.8458815Z  * [new branch]              gh/drisspg/183/head         -> origin/gh/drisspg/183/head
2025-09-07T07:36:03.8460398Z  * [new branch]              gh/drisspg/184/base         -> origin/gh/drisspg/184/base
2025-09-07T07:36:03.8461582Z  * [new branch]              gh/drisspg/184/head         -> origin/gh/drisspg/184/head
2025-09-07T07:36:03.8463318Z  * [new branch]              gh/drisspg/185/base         -> origin/gh/drisspg/185/base
2025-09-07T07:36:03.8464499Z  * [new branch]              gh/drisspg/185/head         -> origin/gh/drisspg/185/head
2025-09-07T07:36:03.8466180Z  * [new branch]              gh/drisspg/186/base         -> origin/gh/drisspg/186/base
2025-09-07T07:36:03.8467385Z  * [new branch]              gh/drisspg/186/head         -> origin/gh/drisspg/186/head
2025-09-07T07:36:03.8468644Z  * [new branch]              gh/drisspg/186/orig         -> origin/gh/drisspg/186/orig
2025-09-07T07:36:03.8470375Z  * [new branch]              gh/drisspg/187/base         -> origin/gh/drisspg/187/base
2025-09-07T07:36:03.8471548Z  * [new branch]              gh/drisspg/187/head         -> origin/gh/drisspg/187/head
2025-09-07T07:36:03.8472742Z  * [new branch]              gh/drisspg/187/orig         -> origin/gh/drisspg/187/orig
2025-09-07T07:36:03.8474496Z  * [new branch]              gh/drisspg/188/base         -> origin/gh/drisspg/188/base
2025-09-07T07:36:03.8475750Z  * [new branch]              gh/drisspg/188/head         -> origin/gh/drisspg/188/head
2025-09-07T07:36:03.8476885Z  * [new branch]              gh/drisspg/188/orig         -> origin/gh/drisspg/188/orig
2025-09-07T07:36:03.8479023Z  * [new branch]              gh/drisspg/189/base         -> origin/gh/drisspg/189/base
2025-09-07T07:36:03.8480291Z  * [new branch]              gh/drisspg/189/head         -> origin/gh/drisspg/189/head
2025-09-07T07:36:03.8481522Z  * [new branch]              gh/drisspg/189/orig         -> origin/gh/drisspg/189/orig
2025-09-07T07:36:03.8483274Z  * [new branch]              gh/drisspg/190/base         -> origin/gh/drisspg/190/base
2025-09-07T07:36:03.8484497Z  * [new branch]              gh/drisspg/190/head         -> origin/gh/drisspg/190/head
2025-09-07T07:36:03.8485900Z  * [new branch]              gh/drisspg/190/orig         -> origin/gh/drisspg/190/orig
2025-09-07T07:36:03.8487643Z  * [new branch]              gh/drisspg/191/base         -> origin/gh/drisspg/191/base
2025-09-07T07:36:03.8488874Z  * [new branch]              gh/drisspg/191/head         -> origin/gh/drisspg/191/head
2025-09-07T07:36:03.8490096Z  * [new branch]              gh/drisspg/191/orig         -> origin/gh/drisspg/191/orig
2025-09-07T07:36:03.8491856Z  * [new branch]              gh/drisspg/192/base         -> origin/gh/drisspg/192/base
2025-09-07T07:36:03.8492967Z  * [new branch]              gh/drisspg/192/head         -> origin/gh/drisspg/192/head
2025-09-07T07:36:03.8494160Z  * [new branch]              gh/drisspg/192/orig         -> origin/gh/drisspg/192/orig
2025-09-07T07:36:03.8495903Z  * [new branch]              gh/drisspg/193/base         -> origin/gh/drisspg/193/base
2025-09-07T07:36:03.8497194Z  * [new branch]              gh/drisspg/193/head         -> origin/gh/drisspg/193/head
2025-09-07T07:36:03.8498479Z  * [new branch]              gh/drisspg/193/orig         -> origin/gh/drisspg/193/orig
2025-09-07T07:36:03.8500175Z  * [new branch]              gh/drisspg/194/base         -> origin/gh/drisspg/194/base
2025-09-07T07:36:03.8501378Z  * [new branch]              gh/drisspg/194/head         -> origin/gh/drisspg/194/head
2025-09-07T07:36:03.8502629Z  * [new branch]              gh/drisspg/194/orig         -> origin/gh/drisspg/194/orig
2025-09-07T07:36:03.8508361Z  * [new branch]              gh/drisspg/195/base         -> origin/gh/drisspg/195/base
2025-09-07T07:36:03.8508909Z  * [new branch]              gh/drisspg/195/head         -> origin/gh/drisspg/195/head
2025-09-07T07:36:03.8510252Z  * [new branch]              gh/drisspg/195/orig         -> origin/gh/drisspg/195/orig
2025-09-07T07:36:03.8512270Z  * [new branch]              gh/drisspg/196/base         -> origin/gh/drisspg/196/base
2025-09-07T07:36:03.8513571Z  * [new branch]              gh/drisspg/196/head         -> origin/gh/drisspg/196/head
2025-09-07T07:36:03.8514779Z  * [new branch]              gh/drisspg/196/orig         -> origin/gh/drisspg/196/orig
2025-09-07T07:36:03.8516627Z  * [new branch]              gh/drisspg/197/base         -> origin/gh/drisspg/197/base
2025-09-07T07:36:03.8518156Z  * [new branch]              gh/drisspg/197/head         -> origin/gh/drisspg/197/head
2025-09-07T07:36:03.8519363Z  * [new branch]              gh/drisspg/197/orig         -> origin/gh/drisspg/197/orig
2025-09-07T07:36:03.8521333Z  * [new branch]              gh/drisspg/198/base         -> origin/gh/drisspg/198/base
2025-09-07T07:36:03.8522552Z  * [new branch]              gh/drisspg/198/head         -> origin/gh/drisspg/198/head
2025-09-07T07:36:03.8523785Z  * [new branch]              gh/drisspg/198/orig         -> origin/gh/drisspg/198/orig
2025-09-07T07:36:03.8525745Z  * [new branch]              gh/drisspg/199/base         -> origin/gh/drisspg/199/base
2025-09-07T07:36:03.8526911Z  * [new branch]              gh/drisspg/199/head         -> origin/gh/drisspg/199/head
2025-09-07T07:36:03.8528246Z  * [new branch]              gh/drisspg/199/orig         -> origin/gh/drisspg/199/orig
2025-09-07T07:36:03.8530752Z  * [new branch]              gh/dsjohns2/1/base          -> origin/gh/dsjohns2/1/base
2025-09-07T07:36:03.8531839Z  * [new branch]              gh/dsjohns2/1/head          -> origin/gh/dsjohns2/1/head
2025-09-07T07:36:03.8534254Z  * [new branch]              gh/eellison/784/base        -> origin/gh/eellison/784/base
2025-09-07T07:36:03.8535453Z  * [new branch]              gh/eellison/784/head        -> origin/gh/eellison/784/head
2025-09-07T07:36:03.8536646Z  * [new branch]              gh/eellison/784/orig        -> origin/gh/eellison/784/orig
2025-09-07T07:36:03.8538867Z  * [new branch]              gh/eellison/785/base        -> origin/gh/eellison/785/base
2025-09-07T07:36:03.8540068Z  * [new branch]              gh/eellison/785/head        -> origin/gh/eellison/785/head
2025-09-07T07:36:03.8541289Z  * [new branch]              gh/eellison/785/orig        -> origin/gh/eellison/785/orig
2025-09-07T07:36:03.8543061Z  * [new branch]              gh/eellison/789/base        -> origin/gh/eellison/789/base
2025-09-07T07:36:03.8544254Z  * [new branch]              gh/eellison/789/head        -> origin/gh/eellison/789/head
2025-09-07T07:36:03.8545429Z  * [new branch]              gh/eellison/789/orig        -> origin/gh/eellison/789/orig
2025-09-07T07:36:03.8547792Z  * [new branch]              gh/eellison/800/base        -> origin/gh/eellison/800/base
2025-09-07T07:36:03.8548994Z  * [new branch]              gh/eellison/800/head        -> origin/gh/eellison/800/head
2025-09-07T07:36:03.8550252Z  * [new branch]              gh/eellison/800/orig        -> origin/gh/eellison/800/orig
2025-09-07T07:36:03.8552014Z  * [new branch]              gh/eellison/801/base        -> origin/gh/eellison/801/base
2025-09-07T07:36:03.8553210Z  * [new branch]              gh/eellison/801/head        -> origin/gh/eellison/801/head
2025-09-07T07:36:03.8554426Z  * [new branch]              gh/eellison/801/orig        -> origin/gh/eellison/801/orig
2025-09-07T07:36:03.8556321Z  * [new branch]              gh/eellison/802/base        -> origin/gh/eellison/802/base
2025-09-07T07:36:03.8557716Z  * [new branch]              gh/eellison/802/head        -> origin/gh/eellison/802/head
2025-09-07T07:36:03.8558932Z  * [new branch]              gh/eellison/802/orig        -> origin/gh/eellison/802/orig
2025-09-07T07:36:03.8560686Z  * [new branch]              gh/eellison/805/base        -> origin/gh/eellison/805/base
2025-09-07T07:36:03.8561851Z  * [new branch]              gh/eellison/805/head        -> origin/gh/eellison/805/head
2025-09-07T07:36:03.8563104Z  * [new branch]              gh/eellison/805/orig        -> origin/gh/eellison/805/orig
2025-09-07T07:36:03.8565029Z  * [new branch]              gh/eellison/808/base        -> origin/gh/eellison/808/base
2025-09-07T07:36:03.8566291Z  * [new branch]              gh/eellison/808/head        -> origin/gh/eellison/808/head
2025-09-07T07:36:03.8567496Z  * [new branch]              gh/eellison/808/orig        -> origin/gh/eellison/808/orig
2025-09-07T07:36:03.8569243Z  * [new branch]              gh/eellison/809/base        -> origin/gh/eellison/809/base
2025-09-07T07:36:03.8570439Z  * [new branch]              gh/eellison/809/head        -> origin/gh/eellison/809/head
2025-09-07T07:36:03.8571612Z  * [new branch]              gh/eellison/809/orig        -> origin/gh/eellison/809/orig
2025-09-07T07:36:03.8573500Z  * [new branch]              gh/eellison/813/base        -> origin/gh/eellison/813/base
2025-09-07T07:36:03.8574743Z  * [new branch]              gh/eellison/813/head        -> origin/gh/eellison/813/head
2025-09-07T07:36:03.8575936Z  * [new branch]              gh/eellison/813/orig        -> origin/gh/eellison/813/orig
2025-09-07T07:36:03.8578201Z  * [new branch]              gh/eellison/814/base        -> origin/gh/eellison/814/base
2025-09-07T07:36:03.8579041Z  * [new branch]              gh/eellison/814/head        -> origin/gh/eellison/814/head
2025-09-07T07:36:03.8580264Z  * [new branch]              gh/eellison/814/orig        -> origin/gh/eellison/814/orig
2025-09-07T07:36:03.8582730Z  * [new branch]              gh/eellison/815/base        -> origin/gh/eellison/815/base
2025-09-07T07:36:03.8583656Z  * [new branch]              gh/eellison/815/head        -> origin/gh/eellison/815/head
2025-09-07T07:36:03.8584951Z  * [new branch]              gh/eellison/815/orig        -> origin/gh/eellison/815/orig
2025-09-07T07:36:03.8586680Z  * [new branch]              gh/eellison/816/base        -> origin/gh/eellison/816/base
2025-09-07T07:36:03.8587845Z  * [new branch]              gh/eellison/816/head        -> origin/gh/eellison/816/head
2025-09-07T07:36:03.8589050Z  * [new branch]              gh/eellison/816/orig        -> origin/gh/eellison/816/orig
2025-09-07T07:36:03.8590891Z  * [new branch]              gh/eellison/817/base        -> origin/gh/eellison/817/base
2025-09-07T07:36:03.8592107Z  * [new branch]              gh/eellison/817/head        -> origin/gh/eellison/817/head
2025-09-07T07:36:03.8593379Z  * [new branch]              gh/eellison/817/orig        -> origin/gh/eellison/817/orig
2025-09-07T07:36:03.8595277Z  * [new branch]              gh/eellison/818/base        -> origin/gh/eellison/818/base
2025-09-07T07:36:03.8596400Z  * [new branch]              gh/eellison/818/head        -> origin/gh/eellison/818/head
2025-09-07T07:36:03.8597583Z  * [new branch]              gh/eellison/818/orig        -> origin/gh/eellison/818/orig
2025-09-07T07:36:03.8599539Z  * [new branch]              gh/eellison/819/base        -> origin/gh/eellison/819/base
2025-09-07T07:36:03.8600683Z  * [new branch]              gh/eellison/819/head        -> origin/gh/eellison/819/head
2025-09-07T07:36:03.8601855Z  * [new branch]              gh/eellison/819/orig        -> origin/gh/eellison/819/orig
2025-09-07T07:36:03.8604501Z  * [new branch]              gh/eellison/820/base        -> origin/gh/eellison/820/base
2025-09-07T07:36:03.8605807Z  * [new branch]              gh/eellison/820/head        -> origin/gh/eellison/820/head
2025-09-07T07:36:03.8607014Z  * [new branch]              gh/eellison/820/orig        -> origin/gh/eellison/820/orig
2025-09-07T07:36:03.8608858Z  * [new branch]              gh/eellison/821/base        -> origin/gh/eellison/821/base
2025-09-07T07:36:03.8610084Z  * [new branch]              gh/eellison/821/head        -> origin/gh/eellison/821/head
2025-09-07T07:36:03.8611304Z  * [new branch]              gh/eellison/821/orig        -> origin/gh/eellison/821/orig
2025-09-07T07:36:03.8613055Z  * [new branch]              gh/eellison/822/base        -> origin/gh/eellison/822/base
2025-09-07T07:36:03.8614232Z  * [new branch]              gh/eellison/822/head        -> origin/gh/eellison/822/head
2025-09-07T07:36:03.8615449Z  * [new branch]              gh/eellison/822/orig        -> origin/gh/eellison/822/orig
2025-09-07T07:36:03.8617313Z  * [new branch]              gh/eellison/823/base        -> origin/gh/eellison/823/base
2025-09-07T07:36:03.8618587Z  * [new branch]              gh/eellison/823/head        -> origin/gh/eellison/823/head
2025-09-07T07:36:03.8619876Z  * [new branch]              gh/eellison/823/orig        -> origin/gh/eellison/823/orig
2025-09-07T07:36:03.8622027Z  * [new branch]              gh/etaf/132/base            -> origin/gh/etaf/132/base
2025-09-07T07:36:03.8623231Z  * [new branch]              gh/etaf/132/head            -> origin/gh/etaf/132/head
2025-09-07T07:36:03.8624514Z  * [new branch]              gh/etaf/132/orig            -> origin/gh/etaf/132/orig
2025-09-07T07:36:03.8626222Z  * [new branch]              gh/etaf/138/base            -> origin/gh/etaf/138/base
2025-09-07T07:36:03.8627405Z  * [new branch]              gh/etaf/138/head            -> origin/gh/etaf/138/head
2025-09-07T07:36:03.8629021Z  * [new branch]              gh/etaf/138/orig            -> origin/gh/etaf/138/orig
2025-09-07T07:36:03.8630936Z  * [new branch]              gh/etaf/140/base            -> origin/gh/etaf/140/base
2025-09-07T07:36:03.8632108Z  * [new branch]              gh/etaf/140/head            -> origin/gh/etaf/140/head
2025-09-07T07:36:03.8633432Z  * [new branch]              gh/etaf/140/orig            -> origin/gh/etaf/140/orig
2025-09-07T07:36:03.8635066Z  * [new branch]              gh/etaf/143/base            -> origin/gh/etaf/143/base
2025-09-07T07:36:03.8636272Z  * [new branch]              gh/etaf/143/head            -> origin/gh/etaf/143/head
2025-09-07T07:36:03.8637470Z  * [new branch]              gh/etaf/143/orig            -> origin/gh/etaf/143/orig
2025-09-07T07:36:03.8639180Z  * [new branch]              gh/etaf/147/base            -> origin/gh/etaf/147/base
2025-09-07T07:36:03.8640399Z  * [new branch]              gh/etaf/147/head            -> origin/gh/etaf/147/head
2025-09-07T07:36:03.8642250Z  * [new branch]              gh/etaf/151/base            -> origin/gh/etaf/151/base
2025-09-07T07:36:03.8643532Z  * [new branch]              gh/etaf/151/head            -> origin/gh/etaf/151/head
2025-09-07T07:36:03.8644745Z  * [new branch]              gh/etaf/151/orig            -> origin/gh/etaf/151/orig
2025-09-07T07:36:03.8646524Z  * [new branch]              gh/etaf/152/base            -> origin/gh/etaf/152/base
2025-09-07T07:36:03.8648234Z  * [new branch]              gh/etaf/152/head            -> origin/gh/etaf/152/head
2025-09-07T07:36:03.8649437Z  * [new branch]              gh/etaf/152/orig            -> origin/gh/etaf/152/orig
2025-09-07T07:36:03.8651686Z  * [new branch]              gh/etaf/153/base            -> origin/gh/etaf/153/base
2025-09-07T07:36:03.8652935Z  * [new branch]              gh/etaf/153/head            -> origin/gh/etaf/153/head
2025-09-07T07:36:03.8654137Z  * [new branch]              gh/etaf/153/orig            -> origin/gh/etaf/153/orig
2025-09-07T07:36:03.8656021Z  * [new branch]              gh/etaf/154/base            -> origin/gh/etaf/154/base
2025-09-07T07:36:03.8657250Z  * [new branch]              gh/etaf/154/head            -> origin/gh/etaf/154/head
2025-09-07T07:36:03.8658561Z  * [new branch]              gh/etaf/154/orig            -> origin/gh/etaf/154/orig
2025-09-07T07:36:03.8660399Z  * [new branch]              gh/etaf/155/base            -> origin/gh/etaf/155/base
2025-09-07T07:36:03.8661679Z  * [new branch]              gh/etaf/155/head            -> origin/gh/etaf/155/head
2025-09-07T07:36:03.8662848Z  * [new branch]              gh/etaf/155/orig            -> origin/gh/etaf/155/orig
2025-09-07T07:36:03.8664500Z  * [new branch]              gh/etaf/156/base            -> origin/gh/etaf/156/base
2025-09-07T07:36:03.8665800Z  * [new branch]              gh/etaf/156/head            -> origin/gh/etaf/156/head
2025-09-07T07:36:03.8667139Z  * [new branch]              gh/etaf/156/orig            -> origin/gh/etaf/156/orig
2025-09-07T07:36:03.8668977Z  * [new branch]              gh/etaf/157/base            -> origin/gh/etaf/157/base
2025-09-07T07:36:03.8670192Z  * [new branch]              gh/etaf/157/head            -> origin/gh/etaf/157/head
2025-09-07T07:36:03.8671409Z  * [new branch]              gh/etaf/157/orig            -> origin/gh/etaf/157/orig
2025-09-07T07:36:03.8673019Z  * [new branch]              gh/etaf/158/base            -> origin/gh/etaf/158/base
2025-09-07T07:36:03.8674259Z  * [new branch]              gh/etaf/158/head            -> origin/gh/etaf/158/head
2025-09-07T07:36:03.8675491Z  * [new branch]              gh/etaf/158/orig            -> origin/gh/etaf/158/orig
2025-09-07T07:36:03.8677310Z  * [new branch]              gh/etaf/159/base            -> origin/gh/etaf/159/base
2025-09-07T07:36:03.8678570Z  * [new branch]              gh/etaf/159/head            -> origin/gh/etaf/159/head
2025-09-07T07:36:03.8679742Z  * [new branch]              gh/etaf/159/orig            -> origin/gh/etaf/159/orig
2025-09-07T07:36:03.8681933Z  * [new branch]              gh/etaf/160/base            -> origin/gh/etaf/160/base
2025-09-07T07:36:03.8683186Z  * [new branch]              gh/etaf/160/head            -> origin/gh/etaf/160/head
2025-09-07T07:36:03.8684418Z  * [new branch]              gh/etaf/160/orig            -> origin/gh/etaf/160/orig
2025-09-07T07:36:03.8686210Z  * [new branch]              gh/etaf/161/base            -> origin/gh/etaf/161/base
2025-09-07T07:36:03.8687488Z  * [new branch]              gh/etaf/161/head            -> origin/gh/etaf/161/head
2025-09-07T07:36:03.8688646Z  * [new branch]              gh/etaf/161/orig            -> origin/gh/etaf/161/orig
2025-09-07T07:36:03.8690800Z  * [new branch]              gh/etaf/162/base            -> origin/gh/etaf/162/base
2025-09-07T07:36:03.8692027Z  * [new branch]              gh/etaf/162/head            -> origin/gh/etaf/162/head
2025-09-07T07:36:03.8693227Z  * [new branch]              gh/etaf/162/orig            -> origin/gh/etaf/162/orig
2025-09-07T07:36:03.8695059Z  * [new branch]              gh/etaf/163/base            -> origin/gh/etaf/163/base
2025-09-07T07:36:03.8696256Z  * [new branch]              gh/etaf/163/head            -> origin/gh/etaf/163/head
2025-09-07T07:36:03.8697418Z  * [new branch]              gh/etaf/163/orig            -> origin/gh/etaf/163/orig
2025-09-07T07:36:03.8699345Z  * [new branch]              gh/etaf/164/base            -> origin/gh/etaf/164/base
2025-09-07T07:36:03.8700603Z  * [new branch]              gh/etaf/164/head            -> origin/gh/etaf/164/head
2025-09-07T07:36:03.8702206Z  * [new branch]              gh/etaf/164/orig            -> origin/gh/etaf/164/orig
2025-09-07T07:36:03.8704428Z  * [new branch]              gh/etaf/165/base            -> origin/gh/etaf/165/base
2025-09-07T07:36:03.8705624Z  * [new branch]              gh/etaf/165/orig            -> origin/gh/etaf/165/orig
2025-09-07T07:36:03.8707875Z  * [new branch]              gh/etaf/166/base            -> origin/gh/etaf/166/base
2025-09-07T07:36:03.8709096Z  * [new branch]              gh/etaf/166/head            -> origin/gh/etaf/166/head
2025-09-07T07:36:03.8710305Z  * [new branch]              gh/etaf/166/orig            -> origin/gh/etaf/166/orig
2025-09-07T07:36:03.8712232Z  * [new branch]              gh/etaf/167/base            -> origin/gh/etaf/167/base
2025-09-07T07:36:03.8713503Z  * [new branch]              gh/etaf/167/head            -> origin/gh/etaf/167/head
2025-09-07T07:36:03.8714704Z  * [new branch]              gh/etaf/167/orig            -> origin/gh/etaf/167/orig
2025-09-07T07:36:03.8716472Z  * [new branch]              gh/etaf/168/base            -> origin/gh/etaf/168/base
2025-09-07T07:36:03.8717735Z  * [new branch]              gh/etaf/168/head            -> origin/gh/etaf/168/head
2025-09-07T07:36:03.8718968Z  * [new branch]              gh/etaf/168/orig            -> origin/gh/etaf/168/orig
2025-09-07T07:36:03.8720841Z  * [new branch]              gh/etaf/169/base            -> origin/gh/etaf/169/base
2025-09-07T07:36:03.8722041Z  * [new branch]              gh/etaf/169/head            -> origin/gh/etaf/169/head
2025-09-07T07:36:03.8723433Z  * [new branch]              gh/etaf/169/orig            -> origin/gh/etaf/169/orig
2025-09-07T07:36:03.8725423Z  * [new branch]              gh/exclamaforte/1/base      -> origin/gh/exclamaforte/1/base
2025-09-07T07:36:03.8726632Z  * [new branch]              gh/exclamaforte/1/head      -> origin/gh/exclamaforte/1/head
2025-09-07T07:36:03.8728320Z  * [new branch]              gh/exclamaforte/2/base      -> origin/gh/exclamaforte/2/base
2025-09-07T07:36:03.8729451Z  * [new branch]              gh/exclamaforte/2/head      -> origin/gh/exclamaforte/2/head
2025-09-07T07:36:03.8731356Z  * [new branch]              gh/exclamaforte/3/base      -> origin/gh/exclamaforte/3/base
2025-09-07T07:36:03.8732398Z  * [new branch]              gh/exclamaforte/3/head      -> origin/gh/exclamaforte/3/head
2025-09-07T07:36:03.8734054Z  * [new branch]              gh/exclamaforte/4/base      -> origin/gh/exclamaforte/4/base
2025-09-07T07:36:03.8735206Z  * [new branch]              gh/exclamaforte/4/head      -> origin/gh/exclamaforte/4/head
2025-09-07T07:36:03.8737573Z  * [new branch]              gh/ezyang/2374/base         -> origin/gh/ezyang/2374/base
2025-09-07T07:36:03.8738940Z  * [new branch]              gh/ezyang/2374/head         -> origin/gh/ezyang/2374/head
2025-09-07T07:36:03.8740280Z  * [new branch]              gh/ezyang/2374/orig         -> origin/gh/ezyang/2374/orig
2025-09-07T07:36:03.8742072Z  * [new branch]              gh/ezyang/2973/base         -> origin/gh/ezyang/2973/base
2025-09-07T07:36:03.8743395Z  * [new branch]              gh/ezyang/2973/head         -> origin/gh/ezyang/2973/head
2025-09-07T07:36:03.8744711Z  * [new branch]              gh/ezyang/2973/orig         -> origin/gh/ezyang/2973/orig
2025-09-07T07:36:03.8746578Z  * [new branch]              gh/ezyang/2974/base         -> origin/gh/ezyang/2974/base
2025-09-07T07:36:03.8747782Z  * [new branch]              gh/ezyang/2974/head         -> origin/gh/ezyang/2974/head
2025-09-07T07:36:03.8749034Z  * [new branch]              gh/ezyang/2974/orig         -> origin/gh/ezyang/2974/orig
2025-09-07T07:36:03.8750780Z  * [new branch]              gh/ezyang/3074/base         -> origin/gh/ezyang/3074/base
2025-09-07T07:36:03.8751936Z  * [new branch]              gh/ezyang/3074/head         -> origin/gh/ezyang/3074/head
2025-09-07T07:36:03.8753179Z  * [new branch]              gh/ezyang/3074/orig         -> origin/gh/ezyang/3074/orig
2025-09-07T07:36:03.8754858Z  * [new branch]              gh/ezyang/3088/base         -> origin/gh/ezyang/3088/base
2025-09-07T07:36:03.8756036Z  * [new branch]              gh/ezyang/3088/head         -> origin/gh/ezyang/3088/head
2025-09-07T07:36:03.8757338Z  * [new branch]              gh/ezyang/3088/orig         -> origin/gh/ezyang/3088/orig
2025-09-07T07:36:03.8759020Z  * [new branch]              gh/ezyang/3092/base         -> origin/gh/ezyang/3092/base
2025-09-07T07:36:03.8760706Z  * [new branch]              gh/ezyang/3092/head         -> origin/gh/ezyang/3092/head
2025-09-07T07:36:03.8761462Z  * [new branch]              gh/ezyang/3092/orig         -> origin/gh/ezyang/3092/orig
2025-09-07T07:36:03.8763198Z  * [new branch]              gh/ezyang/3103/base         -> origin/gh/ezyang/3103/base
2025-09-07T07:36:03.8764385Z  * [new branch]              gh/ezyang/3103/head         -> origin/gh/ezyang/3103/head
2025-09-07T07:36:03.8765555Z  * [new branch]              gh/ezyang/3103/orig         -> origin/gh/ezyang/3103/orig
2025-09-07T07:36:03.8767254Z  * [new branch]              gh/ezyang/3105/base         -> origin/gh/ezyang/3105/base
2025-09-07T07:36:03.8768417Z  * [new branch]              gh/ezyang/3105/head         -> origin/gh/ezyang/3105/head
2025-09-07T07:36:03.8769610Z  * [new branch]              gh/ezyang/3105/orig         -> origin/gh/ezyang/3105/orig
2025-09-07T07:36:03.8771365Z  * [new branch]              gh/ezyang/3114/base         -> origin/gh/ezyang/3114/base
2025-09-07T07:36:03.8772607Z  * [new branch]              gh/ezyang/3114/head         -> origin/gh/ezyang/3114/head
2025-09-07T07:36:03.8773827Z  * [new branch]              gh/ezyang/3114/orig         -> origin/gh/ezyang/3114/orig
2025-09-07T07:36:03.8775480Z  * [new branch]              gh/ezyang/3116/base         -> origin/gh/ezyang/3116/base
2025-09-07T07:36:03.8776633Z  * [new branch]              gh/ezyang/3116/head         -> origin/gh/ezyang/3116/head
2025-09-07T07:36:03.8778135Z  * [new branch]              gh/ezyang/3116/orig         -> origin/gh/ezyang/3116/orig
2025-09-07T07:36:03.8779827Z  * [new branch]              gh/ezyang/3120/base         -> origin/gh/ezyang/3120/base
2025-09-07T07:36:03.8781007Z  * [new branch]              gh/ezyang/3120/head         -> origin/gh/ezyang/3120/head
2025-09-07T07:36:03.8782174Z  * [new branch]              gh/ezyang/3120/orig         -> origin/gh/ezyang/3120/orig
2025-09-07T07:36:03.8783938Z  * [new branch]              gh/ezyang/3122/base         -> origin/gh/ezyang/3122/base
2025-09-07T07:36:03.8785170Z  * [new branch]              gh/ezyang/3122/head         -> origin/gh/ezyang/3122/head
2025-09-07T07:36:03.8786400Z  * [new branch]              gh/ezyang/3122/orig         -> origin/gh/ezyang/3122/orig
2025-09-07T07:36:03.8788115Z  * [new branch]              gh/ezyang/3123/base         -> origin/gh/ezyang/3123/base
2025-09-07T07:36:03.8789319Z  * [new branch]              gh/ezyang/3123/head         -> origin/gh/ezyang/3123/head
2025-09-07T07:36:03.8790586Z  * [new branch]              gh/ezyang/3123/orig         -> origin/gh/ezyang/3123/orig
2025-09-07T07:36:03.8792208Z  * [new branch]              gh/ezyang/3125/base         -> origin/gh/ezyang/3125/base
2025-09-07T07:36:03.8793423Z  * [new branch]              gh/ezyang/3125/head         -> origin/gh/ezyang/3125/head
2025-09-07T07:36:03.8794679Z  * [new branch]              gh/ezyang/3125/orig         -> origin/gh/ezyang/3125/orig
2025-09-07T07:36:03.8796332Z  * [new branch]              gh/ezyang/3126/base         -> origin/gh/ezyang/3126/base
2025-09-07T07:36:03.8797499Z  * [new branch]              gh/ezyang/3126/head         -> origin/gh/ezyang/3126/head
2025-09-07T07:36:03.8798659Z  * [new branch]              gh/ezyang/3126/orig         -> origin/gh/ezyang/3126/orig
2025-09-07T07:36:03.8800847Z  * [new branch]              gh/ezyang/3127/base         -> origin/gh/ezyang/3127/base
2025-09-07T07:36:03.8802064Z  * [new branch]              gh/ezyang/3127/head         -> origin/gh/ezyang/3127/head
2025-09-07T07:36:03.8803402Z  * [new branch]              gh/ezyang/3127/orig         -> origin/gh/ezyang/3127/orig
2025-09-07T07:36:03.8805274Z  * [new branch]              gh/ezyang/3128/base         -> origin/gh/ezyang/3128/base
2025-09-07T07:36:03.8806468Z  * [new branch]              gh/ezyang/3128/head         -> origin/gh/ezyang/3128/head
2025-09-07T07:36:03.8807633Z  * [new branch]              gh/ezyang/3128/orig         -> origin/gh/ezyang/3128/orig
2025-09-07T07:36:03.8809366Z  * [new branch]              gh/ezyang/3129/base         -> origin/gh/ezyang/3129/base
2025-09-07T07:36:03.8810535Z  * [new branch]              gh/ezyang/3129/head         -> origin/gh/ezyang/3129/head
2025-09-07T07:36:03.8811793Z  * [new branch]              gh/ezyang/3129/orig         -> origin/gh/ezyang/3129/orig
2025-09-07T07:36:03.8813700Z  * [new branch]              gh/ezyang/3130/base         -> origin/gh/ezyang/3130/base
2025-09-07T07:36:03.8814902Z  * [new branch]              gh/ezyang/3130/head         -> origin/gh/ezyang/3130/head
2025-09-07T07:36:03.8816096Z  * [new branch]              gh/ezyang/3130/orig         -> origin/gh/ezyang/3130/orig
2025-09-07T07:36:03.8817869Z  * [new branch]              gh/ezyang/3131/base         -> origin/gh/ezyang/3131/base
2025-09-07T07:36:03.8819133Z  * [new branch]              gh/ezyang/3131/head         -> origin/gh/ezyang/3131/head
2025-09-07T07:36:03.8820366Z  * [new branch]              gh/ezyang/3131/orig         -> origin/gh/ezyang/3131/orig
2025-09-07T07:36:03.8822143Z  * [new branch]              gh/ezyang/3132/base         -> origin/gh/ezyang/3132/base
2025-09-07T07:36:03.8823299Z  * [new branch]              gh/ezyang/3132/head         -> origin/gh/ezyang/3132/head
2025-09-07T07:36:03.8824476Z  * [new branch]              gh/ezyang/3132/orig         -> origin/gh/ezyang/3132/orig
2025-09-07T07:36:03.8826180Z  * [new branch]              gh/ezyang/3133/base         -> origin/gh/ezyang/3133/base
2025-09-07T07:36:03.8827362Z  * [new branch]              gh/ezyang/3133/head         -> origin/gh/ezyang/3133/head
2025-09-07T07:36:03.8828546Z  * [new branch]              gh/ezyang/3133/orig         -> origin/gh/ezyang/3133/orig
2025-09-07T07:36:03.8830383Z  * [new branch]              gh/ezyang/3134/base         -> origin/gh/ezyang/3134/base
2025-09-07T07:36:03.8831544Z  * [new branch]              gh/ezyang/3134/head         -> origin/gh/ezyang/3134/head
2025-09-07T07:36:03.8832691Z  * [new branch]              gh/ezyang/3134/orig         -> origin/gh/ezyang/3134/orig
2025-09-07T07:36:03.8834495Z  * [new branch]              gh/ezyang/3135/base         -> origin/gh/ezyang/3135/base
2025-09-07T07:36:03.8835673Z  * [new branch]              gh/ezyang/3135/head         -> origin/gh/ezyang/3135/head
2025-09-07T07:36:03.8836960Z  * [new branch]              gh/ezyang/3135/orig         -> origin/gh/ezyang/3135/orig
2025-09-07T07:36:03.8838663Z  * [new branch]              gh/ezyang/3136/base         -> origin/gh/ezyang/3136/base
2025-09-07T07:36:03.8839917Z  * [new branch]              gh/ezyang/3136/head         -> origin/gh/ezyang/3136/head
2025-09-07T07:36:03.8841018Z  * [new branch]              gh/ezyang/3136/orig         -> origin/gh/ezyang/3136/orig
2025-09-07T07:36:03.8842769Z  * [new branch]              gh/ezyang/3137/base         -> origin/gh/ezyang/3137/base
2025-09-07T07:36:03.8844102Z  * [new branch]              gh/ezyang/3137/head         -> origin/gh/ezyang/3137/head
2025-09-07T07:36:03.8845259Z  * [new branch]              gh/ezyang/3137/orig         -> origin/gh/ezyang/3137/orig
2025-09-07T07:36:03.8847126Z  * [new branch]              gh/ezyang/3138/base         -> origin/gh/ezyang/3138/base
2025-09-07T07:36:03.8848600Z  * [new branch]              gh/ezyang/3138/head         -> origin/gh/ezyang/3138/head
2025-09-07T07:36:03.8849961Z  * [new branch]              gh/ezyang/3138/orig         -> origin/gh/ezyang/3138/orig
2025-09-07T07:36:03.8851807Z  * [new branch]              gh/ezyang/3139/base         -> origin/gh/ezyang/3139/base
2025-09-07T07:36:03.8852977Z  * [new branch]              gh/ezyang/3139/head         -> origin/gh/ezyang/3139/head
2025-09-07T07:36:03.8854200Z  * [new branch]              gh/ezyang/3139/orig         -> origin/gh/ezyang/3139/orig
2025-09-07T07:36:03.8855957Z  * [new branch]              gh/ezyang/3140/base         -> origin/gh/ezyang/3140/base
2025-09-07T07:36:03.8857127Z  * [new branch]              gh/ezyang/3140/head         -> origin/gh/ezyang/3140/head
2025-09-07T07:36:03.8858478Z  * [new branch]              gh/ezyang/3140/orig         -> origin/gh/ezyang/3140/orig
2025-09-07T07:36:03.8860223Z  * [new branch]              gh/ezyang/3141/base         -> origin/gh/ezyang/3141/base
2025-09-07T07:36:03.8861423Z  * [new branch]              gh/ezyang/3141/head         -> origin/gh/ezyang/3141/head
2025-09-07T07:36:03.8862624Z  * [new branch]              gh/ezyang/3141/orig         -> origin/gh/ezyang/3141/orig
2025-09-07T07:36:03.8864434Z  * [new branch]              gh/ezyang/3142/base         -> origin/gh/ezyang/3142/base
2025-09-07T07:36:03.8865598Z  * [new branch]              gh/ezyang/3142/head         -> origin/gh/ezyang/3142/head
2025-09-07T07:36:03.8866778Z  * [new branch]              gh/ezyang/3142/orig         -> origin/gh/ezyang/3142/orig
2025-09-07T07:36:03.8868486Z  * [new branch]              gh/ezyang/3143/base         -> origin/gh/ezyang/3143/base
2025-09-07T07:36:03.8869679Z  * [new branch]              gh/ezyang/3143/head         -> origin/gh/ezyang/3143/head
2025-09-07T07:36:03.8870859Z  * [new branch]              gh/ezyang/3143/orig         -> origin/gh/ezyang/3143/orig
2025-09-07T07:36:03.8872975Z  * [new branch]              gh/fadara01/1/base          -> origin/gh/fadara01/1/base
2025-09-07T07:36:03.8875177Z  * [new branch]              gh/fadara01/1/head          -> origin/gh/fadara01/1/head
2025-09-07T07:36:03.8876385Z  * [new branch]              gh/fadara01/1/orig          -> origin/gh/fadara01/1/orig
2025-09-07T07:36:03.8878652Z  * [new branch]              gh/fduwjj/171/base          -> origin/gh/fduwjj/171/base
2025-09-07T07:36:03.8879932Z  * [new branch]              gh/fduwjj/171/head          -> origin/gh/fduwjj/171/head
2025-09-07T07:36:03.8881181Z  * [new branch]              gh/fduwjj/171/orig          -> origin/gh/fduwjj/171/orig
2025-09-07T07:36:03.8882999Z  * [new branch]              gh/fduwjj/175/base          -> origin/gh/fduwjj/175/base
2025-09-07T07:36:03.8884325Z  * [new branch]              gh/fduwjj/175/head          -> origin/gh/fduwjj/175/head
2025-09-07T07:36:03.8885638Z  * [new branch]              gh/fduwjj/175/orig          -> origin/gh/fduwjj/175/orig
2025-09-07T07:36:03.8887461Z  * [new branch]              gh/fduwjj/176/base          -> origin/gh/fduwjj/176/base
2025-09-07T07:36:03.8888635Z  * [new branch]              gh/fduwjj/176/head          -> origin/gh/fduwjj/176/head
2025-09-07T07:36:03.8889847Z  * [new branch]              gh/fduwjj/176/orig          -> origin/gh/fduwjj/176/orig
2025-09-07T07:36:03.8891608Z  * [new branch]              gh/fduwjj/177/base          -> origin/gh/fduwjj/177/base
2025-09-07T07:36:03.8892900Z  * [new branch]              gh/fduwjj/177/head          -> origin/gh/fduwjj/177/head
2025-09-07T07:36:03.8894019Z  * [new branch]              gh/fduwjj/177/orig          -> origin/gh/fduwjj/177/orig
2025-09-07T07:36:03.8895703Z  * [new branch]              gh/fduwjj/178/base          -> origin/gh/fduwjj/178/base
2025-09-07T07:36:03.8896976Z  * [new branch]              gh/fduwjj/178/head          -> origin/gh/fduwjj/178/head
2025-09-07T07:36:03.8898330Z  * [new branch]              gh/fduwjj/178/orig          -> origin/gh/fduwjj/178/orig
2025-09-07T07:36:03.8900044Z  * [new branch]              gh/fduwjj/179/base          -> origin/gh/fduwjj/179/base
2025-09-07T07:36:03.8901188Z  * [new branch]              gh/fduwjj/179/head          -> origin/gh/fduwjj/179/head
2025-09-07T07:36:03.8902394Z  * [new branch]              gh/fduwjj/179/orig          -> origin/gh/fduwjj/179/orig
2025-09-07T07:36:03.8904329Z  * [new branch]              gh/fduwjj/180/base          -> origin/gh/fduwjj/180/base
2025-09-07T07:36:03.8905537Z  * [new branch]              gh/fduwjj/180/head          -> origin/gh/fduwjj/180/head
2025-09-07T07:36:03.8906722Z  * [new branch]              gh/fduwjj/180/orig          -> origin/gh/fduwjj/180/orig
2025-09-07T07:36:03.8908428Z  * [new branch]              gh/fduwjj/181/base          -> origin/gh/fduwjj/181/base
2025-09-07T07:36:03.8909598Z  * [new branch]              gh/fduwjj/181/head          -> origin/gh/fduwjj/181/head
2025-09-07T07:36:03.8910787Z  * [new branch]              gh/fduwjj/181/orig          -> origin/gh/fduwjj/181/orig
2025-09-07T07:36:03.8912473Z  * [new branch]              gh/fduwjj/182/base          -> origin/gh/fduwjj/182/base
2025-09-07T07:36:03.8913655Z  * [new branch]              gh/fduwjj/182/head          -> origin/gh/fduwjj/182/head
2025-09-07T07:36:03.8914913Z  * [new branch]              gh/fduwjj/182/orig          -> origin/gh/fduwjj/182/orig
2025-09-07T07:36:03.8916680Z  * [new branch]              gh/fduwjj/183/base          -> origin/gh/fduwjj/183/base
2025-09-07T07:36:03.8918021Z  * [new branch]              gh/fduwjj/183/head          -> origin/gh/fduwjj/183/head
2025-09-07T07:36:03.8919220Z  * [new branch]              gh/fduwjj/183/orig          -> origin/gh/fduwjj/183/orig
2025-09-07T07:36:03.8921293Z  * [new branch]              gh/fduwjj/184/base          -> origin/gh/fduwjj/184/base
2025-09-07T07:36:03.8922468Z  * [new branch]              gh/fduwjj/184/head          -> origin/gh/fduwjj/184/head
2025-09-07T07:36:03.8923654Z  * [new branch]              gh/fduwjj/184/orig          -> origin/gh/fduwjj/184/orig
2025-09-07T07:36:03.8925436Z  * [new branch]              gh/fduwjj/185/base          -> origin/gh/fduwjj/185/base
2025-09-07T07:36:03.8926603Z  * [new branch]              gh/fduwjj/185/head          -> origin/gh/fduwjj/185/head
2025-09-07T07:36:03.8927812Z  * [new branch]              gh/fduwjj/185/orig          -> origin/gh/fduwjj/185/orig
2025-09-07T07:36:03.8929406Z  * [new branch]              gh/fduwjj/186/base          -> origin/gh/fduwjj/186/base
2025-09-07T07:36:03.8930610Z  * [new branch]              gh/fduwjj/186/head          -> origin/gh/fduwjj/186/head
2025-09-07T07:36:03.8931796Z  * [new branch]              gh/fduwjj/186/orig          -> origin/gh/fduwjj/186/orig
2025-09-07T07:36:03.8933471Z  * [new branch]              gh/fduwjj/187/base          -> origin/gh/fduwjj/187/base
2025-09-07T07:36:03.8934742Z  * [new branch]              gh/fduwjj/187/head          -> origin/gh/fduwjj/187/head
2025-09-07T07:36:03.8935931Z  * [new branch]              gh/fduwjj/187/orig          -> origin/gh/fduwjj/187/orig
2025-09-07T07:36:03.8937569Z  * [new branch]              gh/fduwjj/188/base          -> origin/gh/fduwjj/188/base
2025-09-07T07:36:03.8938883Z  * [new branch]              gh/fduwjj/188/head          -> origin/gh/fduwjj/188/head
2025-09-07T07:36:03.8940031Z  * [new branch]              gh/fduwjj/188/orig          -> origin/gh/fduwjj/188/orig
2025-09-07T07:36:03.8941750Z  * [new branch]              gh/fduwjj/189/base          -> origin/gh/fduwjj/189/base
2025-09-07T07:36:03.8942759Z  * [new branch]              gh/fduwjj/189/head          -> origin/gh/fduwjj/189/head
2025-09-07T07:36:03.8944388Z  * [new branch]              gh/fduwjj/189/orig          -> origin/gh/fduwjj/189/orig
2025-09-07T07:36:03.8946513Z  * [new branch]              gh/fduwjj/190/base          -> origin/gh/fduwjj/190/base
2025-09-07T07:36:03.8947724Z  * [new branch]              gh/fduwjj/190/head          -> origin/gh/fduwjj/190/head
2025-09-07T07:36:03.8949031Z  * [new branch]              gh/fduwjj/190/orig          -> origin/gh/fduwjj/190/orig
2025-09-07T07:36:03.8950646Z  * [new branch]              gh/fduwjj/191/base          -> origin/gh/fduwjj/191/base
2025-09-07T07:36:03.8952042Z  * [new branch]              gh/fduwjj/191/head          -> origin/gh/fduwjj/191/head
2025-09-07T07:36:03.8953481Z  * [new branch]              gh/fduwjj/191/orig          -> origin/gh/fduwjj/191/orig
2025-09-07T07:36:03.8955750Z  * [new branch]              gh/fegin/306/base           -> origin/gh/fegin/306/base
2025-09-07T07:36:03.8957582Z  * [new branch]              gh/fegin/306/head           -> origin/gh/fegin/306/head
2025-09-07T07:36:03.8958815Z  * [new branch]              gh/fegin/306/orig           -> origin/gh/fegin/306/orig
2025-09-07T07:36:03.8960492Z  * [new branch]              gh/fegin/307/base           -> origin/gh/fegin/307/base
2025-09-07T07:36:03.8961692Z  * [new branch]              gh/fegin/307/head           -> origin/gh/fegin/307/head
2025-09-07T07:36:03.8962894Z  * [new branch]              gh/fegin/307/orig           -> origin/gh/fegin/307/orig
2025-09-07T07:36:03.8964650Z  * [new branch]              gh/fegin/308/base           -> origin/gh/fegin/308/base
2025-09-07T07:36:03.8966725Z  * [new branch]              gh/fegin/308/head           -> origin/gh/fegin/308/head
2025-09-07T07:36:03.8967176Z  * [new branch]              gh/fegin/308/orig           -> origin/gh/fegin/308/orig
2025-09-07T07:36:03.8968858Z  * [new branch]              gh/fegin/309/base           -> origin/gh/fegin/309/base
2025-09-07T07:36:03.8970061Z  * [new branch]              gh/fegin/309/head           -> origin/gh/fegin/309/head
2025-09-07T07:36:03.8971310Z  * [new branch]              gh/fegin/309/orig           -> origin/gh/fegin/309/orig
2025-09-07T07:36:03.8973191Z  * [new branch]              gh/fegin/310/base           -> origin/gh/fegin/310/base
2025-09-07T07:36:03.8974220Z  * [new branch]              gh/fegin/310/head           -> origin/gh/fegin/310/head
2025-09-07T07:36:03.8975455Z  * [new branch]              gh/fegin/310/orig           -> origin/gh/fegin/310/orig
2025-09-07T07:36:03.8977220Z  * [new branch]              gh/fegin/311/base           -> origin/gh/fegin/311/base
2025-09-07T07:36:03.8978518Z  * [new branch]              gh/fegin/311/head           -> origin/gh/fegin/311/head
2025-09-07T07:36:03.8979804Z  * [new branch]              gh/fegin/311/orig           -> origin/gh/fegin/311/orig
2025-09-07T07:36:03.8981439Z  * [new branch]              gh/fegin/312/base           -> origin/gh/fegin/312/base
2025-09-07T07:36:03.8982617Z  * [new branch]              gh/fegin/312/head           -> origin/gh/fegin/312/head
2025-09-07T07:36:03.8983905Z  * [new branch]              gh/fegin/312/orig           -> origin/gh/fegin/312/orig
2025-09-07T07:36:03.8985588Z  * [new branch]              gh/fegin/313/base           -> origin/gh/fegin/313/base
2025-09-07T07:36:03.8986782Z  * [new branch]              gh/fegin/313/head           -> origin/gh/fegin/313/head
2025-09-07T07:36:03.8987971Z  * [new branch]              gh/fegin/313/orig           -> origin/gh/fegin/313/orig
2025-09-07T07:36:03.8990125Z  * [new branch]              gh/fffrog/124/base          -> origin/gh/fffrog/124/base
2025-09-07T07:36:03.8991398Z  * [new branch]              gh/fffrog/124/head          -> origin/gh/fffrog/124/head
2025-09-07T07:36:03.8994796Z  * [new branch]              gh/fffrog/124/orig          -> origin/gh/fffrog/124/orig
2025-09-07T07:36:03.8995762Z  * [new branch]              gh/fffrog/129/base          -> origin/gh/fffrog/129/base
2025-09-07T07:36:03.8996554Z  * [new branch]              gh/fffrog/129/head          -> origin/gh/fffrog/129/head
2025-09-07T07:36:03.8997458Z  * [new branch]              gh/fffrog/129/orig          -> origin/gh/fffrog/129/orig
2025-09-07T07:36:03.8999186Z  * [new branch]              gh/fffrog/130/base          -> origin/gh/fffrog/130/base
2025-09-07T07:36:03.9000447Z  * [new branch]              gh/fffrog/130/head          -> origin/gh/fffrog/130/head
2025-09-07T07:36:03.9001711Z  * [new branch]              gh/fffrog/130/orig          -> origin/gh/fffrog/130/orig
2025-09-07T07:36:03.9003608Z  * [new branch]              gh/fffrog/131/base          -> origin/gh/fffrog/131/base
2025-09-07T07:36:03.9004855Z  * [new branch]              gh/fffrog/131/head          -> origin/gh/fffrog/131/head
2025-09-07T07:36:03.9006074Z  * [new branch]              gh/fffrog/131/orig          -> origin/gh/fffrog/131/orig
2025-09-07T07:36:03.9007742Z  * [new branch]              gh/fffrog/132/base          -> origin/gh/fffrog/132/base
2025-09-07T07:36:03.9008927Z  * [new branch]              gh/fffrog/132/head          -> origin/gh/fffrog/132/head
2025-09-07T07:36:03.9010100Z  * [new branch]              gh/fffrog/132/orig          -> origin/gh/fffrog/132/orig
2025-09-07T07:36:03.9011829Z  * [new branch]              gh/fffrog/133/base          -> origin/gh/fffrog/133/base
2025-09-07T07:36:03.9013007Z  * [new branch]              gh/fffrog/133/head          -> origin/gh/fffrog/133/head
2025-09-07T07:36:03.9014221Z  * [new branch]              gh/fffrog/133/orig          -> origin/gh/fffrog/133/orig
2025-09-07T07:36:03.9015858Z  * [new branch]              gh/fffrog/134/base          -> origin/gh/fffrog/134/base
2025-09-07T07:36:03.9017103Z  * [new branch]              gh/fffrog/134/head          -> origin/gh/fffrog/134/head
2025-09-07T07:36:03.9018838Z  * [new branch]              gh/fffrog/134/orig          -> origin/gh/fffrog/134/orig
2025-09-07T07:36:03.9020670Z  * [new branch]              gh/fffrog/135/base          -> origin/gh/fffrog/135/base
2025-09-07T07:36:03.9021895Z  * [new branch]              gh/fffrog/135/head          -> origin/gh/fffrog/135/head
2025-09-07T07:36:03.9023127Z  * [new branch]              gh/fffrog/135/orig          -> origin/gh/fffrog/135/orig
2025-09-07T07:36:03.9024970Z  * [new branch]              gh/fffrog/136/base          -> origin/gh/fffrog/136/base
2025-09-07T07:36:03.9026009Z  * [new branch]              gh/fffrog/136/head          -> origin/gh/fffrog/136/head
2025-09-07T07:36:03.9027194Z  * [new branch]              gh/fffrog/136/orig          -> origin/gh/fffrog/136/orig
2025-09-07T07:36:03.9029007Z  * [new branch]              gh/fffrog/137/base          -> origin/gh/fffrog/137/base
2025-09-07T07:36:03.9030140Z  * [new branch]              gh/fffrog/137/head          -> origin/gh/fffrog/137/head
2025-09-07T07:36:03.9031347Z  * [new branch]              gh/fffrog/137/orig          -> origin/gh/fffrog/137/orig
2025-09-07T07:36:03.9033009Z  * [new branch]              gh/fffrog/138/base          -> origin/gh/fffrog/138/base
2025-09-07T07:36:03.9034250Z  * [new branch]              gh/fffrog/138/head          -> origin/gh/fffrog/138/head
2025-09-07T07:36:03.9035451Z  * [new branch]              gh/fffrog/138/orig          -> origin/gh/fffrog/138/orig
2025-09-07T07:36:03.9037137Z  * [new branch]              gh/fffrog/139/base          -> origin/gh/fffrog/139/base
2025-09-07T07:36:03.9038356Z  * [new branch]              gh/fffrog/139/head          -> origin/gh/fffrog/139/head
2025-09-07T07:36:03.9039581Z  * [new branch]              gh/fffrog/139/orig          -> origin/gh/fffrog/139/orig
2025-09-07T07:36:03.9041316Z  * [new branch]              gh/fffrog/140/base          -> origin/gh/fffrog/140/base
2025-09-07T07:36:03.9042479Z  * [new branch]              gh/fffrog/140/head          -> origin/gh/fffrog/140/head
2025-09-07T07:36:03.9043611Z  * [new branch]              gh/fffrog/140/orig          -> origin/gh/fffrog/140/orig
2025-09-07T07:36:03.9045375Z  * [new branch]              gh/fffrog/141/base          -> origin/gh/fffrog/141/base
2025-09-07T07:36:03.9046471Z  * [new branch]              gh/fffrog/141/head          -> origin/gh/fffrog/141/head
2025-09-07T07:36:03.9047610Z  * [new branch]              gh/fffrog/141/orig          -> origin/gh/fffrog/141/orig
2025-09-07T07:36:03.9049294Z  * [new branch]              gh/fffrog/142/base          -> origin/gh/fffrog/142/base
2025-09-07T07:36:03.9050611Z  * [new branch]              gh/fffrog/142/head          -> origin/gh/fffrog/142/head
2025-09-07T07:36:03.9051789Z  * [new branch]              gh/fffrog/142/orig          -> origin/gh/fffrog/142/orig
2025-09-07T07:36:03.9053437Z  * [new branch]              gh/fffrog/143/base          -> origin/gh/fffrog/143/base
2025-09-07T07:36:03.9054641Z  * [new branch]              gh/fffrog/143/head          -> origin/gh/fffrog/143/head
2025-09-07T07:36:03.9055837Z  * [new branch]              gh/fffrog/143/orig          -> origin/gh/fffrog/143/orig
2025-09-07T07:36:03.9058053Z  * [new branch]              gh/fffrog/144/base          -> origin/gh/fffrog/144/base
2025-09-07T07:36:03.9059242Z  * [new branch]              gh/fffrog/144/head          -> origin/gh/fffrog/144/head
2025-09-07T07:36:03.9060419Z  * [new branch]              gh/fffrog/144/orig          -> origin/gh/fffrog/144/orig
2025-09-07T07:36:03.9062710Z  * [new branch]              gh/fffrog/145/base          -> origin/gh/fffrog/145/base
2025-09-07T07:36:03.9063403Z  * [new branch]              gh/fffrog/145/head          -> origin/gh/fffrog/145/head
2025-09-07T07:36:03.9064762Z  * [new branch]              gh/fffrog/145/orig          -> origin/gh/fffrog/145/orig
2025-09-07T07:36:03.9066488Z  * [new branch]              gh/fffrog/146/base          -> origin/gh/fffrog/146/base
2025-09-07T07:36:03.9067800Z  * [new branch]              gh/fffrog/146/head          -> origin/gh/fffrog/146/head
2025-09-07T07:36:03.9068973Z  * [new branch]              gh/fffrog/146/orig          -> origin/gh/fffrog/146/orig
2025-09-07T07:36:03.9070761Z  * [new branch]              gh/fffrog/147/base          -> origin/gh/fffrog/147/base
2025-09-07T07:36:03.9072138Z  * [new branch]              gh/fffrog/147/head          -> origin/gh/fffrog/147/head
2025-09-07T07:36:03.9087050Z  * [new branch]              gh/fffrog/147/orig          -> origin/gh/fffrog/147/orig
2025-09-07T07:36:03.9087260Z  * [new branch]              gh/fffrog/148/base          -> origin/gh/fffrog/148/base
2025-09-07T07:36:03.9087410Z  * [new branch]              gh/fffrog/148/head          -> origin/gh/fffrog/148/head
2025-09-07T07:36:03.9087542Z  * [new branch]              gh/fffrog/148/orig          -> origin/gh/fffrog/148/orig
2025-09-07T07:36:03.9087662Z  * [new branch]              gh/fffrog/149/base          -> origin/gh/fffrog/149/base
2025-09-07T07:36:03.9087791Z  * [new branch]              gh/fffrog/149/head          -> origin/gh/fffrog/149/head
2025-09-07T07:36:03.9087911Z  * [new branch]              gh/fffrog/149/orig          -> origin/gh/fffrog/149/orig
2025-09-07T07:36:03.9088030Z  * [new branch]              gh/fffrog/150/base          -> origin/gh/fffrog/150/base
2025-09-07T07:36:03.9088153Z  * [new branch]              gh/fffrog/150/head          -> origin/gh/fffrog/150/head
2025-09-07T07:36:03.9088270Z  * [new branch]              gh/fffrog/150/orig          -> origin/gh/fffrog/150/orig
2025-09-07T07:36:03.9088396Z  * [new branch]              gh/fffrog/151/base          -> origin/gh/fffrog/151/base
2025-09-07T07:36:03.9088519Z  * [new branch]              gh/fffrog/151/head          -> origin/gh/fffrog/151/head
2025-09-07T07:36:03.9089656Z  * [new branch]              gh/fffrog/151/orig          -> origin/gh/fffrog/151/orig
2025-09-07T07:36:03.9091469Z  * [new branch]              gh/fffrog/152/base          -> origin/gh/fffrog/152/base
2025-09-07T07:36:03.9092704Z  * [new branch]              gh/fffrog/152/head          -> origin/gh/fffrog/152/head
2025-09-07T07:36:03.9094548Z  * [new branch]              gh/fffrog/153/base          -> origin/gh/fffrog/153/base
2025-09-07T07:36:03.9095644Z  * [new branch]              gh/fffrog/153/head          -> origin/gh/fffrog/153/head
2025-09-07T07:36:03.9096827Z  * [new branch]              gh/fffrog/153/orig          -> origin/gh/fffrog/153/orig
2025-09-07T07:36:03.9099493Z  * [new branch]              gh/gmagogsfm/1/base         -> origin/gh/gmagogsfm/1/base
2025-09-07T07:36:03.9101002Z  * [new branch]              gh/gmagogsfm/1/head         -> origin/gh/gmagogsfm/1/head
2025-09-07T07:36:03.9102108Z  * [new branch]              gh/gmagogsfm/1/orig         -> origin/gh/gmagogsfm/1/orig
2025-09-07T07:36:03.9103953Z  * [new branch]              gh/gmagogsfm/2/base         -> origin/gh/gmagogsfm/2/base
2025-09-07T07:36:03.9105202Z  * [new branch]              gh/gmagogsfm/2/head         -> origin/gh/gmagogsfm/2/head
2025-09-07T07:36:03.9106384Z  * [new branch]              gh/gmagogsfm/2/orig         -> origin/gh/gmagogsfm/2/orig
2025-09-07T07:36:03.9108002Z  * [new branch]              gh/gmagogsfm/3/base         -> origin/gh/gmagogsfm/3/base
2025-09-07T07:36:03.9109264Z  * [new branch]              gh/gmagogsfm/3/head         -> origin/gh/gmagogsfm/3/head
2025-09-07T07:36:03.9110443Z  * [new branch]              gh/gmagogsfm/3/orig         -> origin/gh/gmagogsfm/3/orig
2025-09-07T07:36:03.9112596Z  * [new branch]              gh/guangyey/134/base        -> origin/gh/guangyey/134/base
2025-09-07T07:36:03.9113772Z  * [new branch]              gh/guangyey/134/head        -> origin/gh/guangyey/134/head
2025-09-07T07:36:03.9114956Z  * [new branch]              gh/guangyey/134/orig        -> origin/gh/guangyey/134/orig
2025-09-07T07:36:03.9116718Z  * [new branch]              gh/guangyey/135/base        -> origin/gh/guangyey/135/base
2025-09-07T07:36:03.9117910Z  * [new branch]              gh/guangyey/135/head        -> origin/gh/guangyey/135/head
2025-09-07T07:36:03.9119087Z  * [new branch]              gh/guangyey/135/orig        -> origin/gh/guangyey/135/orig
2025-09-07T07:36:03.9120786Z  * [new branch]              gh/guangyey/139/base        -> origin/gh/guangyey/139/base
2025-09-07T07:36:03.9122056Z  * [new branch]              gh/guangyey/139/head        -> origin/gh/guangyey/139/head
2025-09-07T07:36:03.9123215Z  * [new branch]              gh/guangyey/139/orig        -> origin/gh/guangyey/139/orig
2025-09-07T07:36:03.9124910Z  * [new branch]              gh/guangyey/140/base        -> origin/gh/guangyey/140/base
2025-09-07T07:36:03.9126083Z  * [new branch]              gh/guangyey/140/head        -> origin/gh/guangyey/140/head
2025-09-07T07:36:03.9127256Z  * [new branch]              gh/guangyey/140/orig        -> origin/gh/guangyey/140/orig
2025-09-07T07:36:03.9128924Z  * [new branch]              gh/guangyey/142/base        -> origin/gh/guangyey/142/base
2025-09-07T07:36:03.9130087Z  * [new branch]              gh/guangyey/142/head        -> origin/gh/guangyey/142/head
2025-09-07T07:36:03.9131289Z  * [new branch]              gh/guangyey/142/orig        -> origin/gh/guangyey/142/orig
2025-09-07T07:36:03.9133025Z  * [new branch]              gh/guangyey/145/base        -> origin/gh/guangyey/145/base
2025-09-07T07:36:03.9134220Z  * [new branch]              gh/guangyey/145/head        -> origin/gh/guangyey/145/head
2025-09-07T07:36:03.9135512Z  * [new branch]              gh/guangyey/145/orig        -> origin/gh/guangyey/145/orig
2025-09-07T07:36:03.9137245Z  * [new branch]              gh/guangyey/153/base        -> origin/gh/guangyey/153/base
2025-09-07T07:36:03.9138471Z  * [new branch]              gh/guangyey/153/head        -> origin/gh/guangyey/153/head
2025-09-07T07:36:03.9139652Z  * [new branch]              gh/guangyey/153/orig        -> origin/gh/guangyey/153/orig
2025-09-07T07:36:03.9141348Z  * [new branch]              gh/guangyey/159/base        -> origin/gh/guangyey/159/base
2025-09-07T07:36:03.9142541Z  * [new branch]              gh/guangyey/159/head        -> origin/gh/guangyey/159/head
2025-09-07T07:36:03.9143825Z  * [new branch]              gh/guangyey/159/orig        -> origin/gh/guangyey/159/orig
2025-09-07T07:36:03.9145521Z  * [new branch]              gh/guangyey/163/base        -> origin/gh/guangyey/163/base
2025-09-07T07:36:03.9146701Z  * [new branch]              gh/guangyey/163/head        -> origin/gh/guangyey/163/head
2025-09-07T07:36:03.9147862Z  * [new branch]              gh/guangyey/163/orig        -> origin/gh/guangyey/163/orig
2025-09-07T07:36:03.9149661Z  * [new branch]              gh/guangyey/168/base        -> origin/gh/guangyey/168/base
2025-09-07T07:36:03.9150842Z  * [new branch]              gh/guangyey/168/head        -> origin/gh/guangyey/168/head
2025-09-07T07:36:03.9152065Z  * [new branch]              gh/guangyey/168/orig        -> origin/gh/guangyey/168/orig
2025-09-07T07:36:03.9153724Z  * [new branch]              gh/guangyey/169/base        -> origin/gh/guangyey/169/base
2025-09-07T07:36:03.9154920Z  * [new branch]              gh/guangyey/169/head        -> origin/gh/guangyey/169/head
2025-09-07T07:36:03.9156082Z  * [new branch]              gh/guangyey/169/orig        -> origin/gh/guangyey/169/orig
2025-09-07T07:36:03.9157792Z  * [new branch]              gh/guangyey/170/base        -> origin/gh/guangyey/170/base
2025-09-07T07:36:03.9158958Z  * [new branch]              gh/guangyey/170/head        -> origin/gh/guangyey/170/head
2025-09-07T07:36:03.9160171Z  * [new branch]              gh/guangyey/170/orig        -> origin/gh/guangyey/170/orig
2025-09-07T07:36:03.9161843Z  * [new branch]              gh/guangyey/171/base        -> origin/gh/guangyey/171/base
2025-09-07T07:36:03.9163050Z  * [new branch]              gh/guangyey/171/head        -> origin/gh/guangyey/171/head
2025-09-07T07:36:03.9164229Z  * [new branch]              gh/guangyey/171/orig        -> origin/gh/guangyey/171/orig
2025-09-07T07:36:03.9166404Z  * [new branch]              gh/guangyey/174/base        -> origin/gh/guangyey/174/base
2025-09-07T07:36:03.9167607Z  * [new branch]              gh/guangyey/174/head        -> origin/gh/guangyey/174/head
2025-09-07T07:36:03.9168776Z  * [new branch]              gh/guangyey/174/orig        -> origin/gh/guangyey/174/orig
2025-09-07T07:36:03.9170598Z  * [new branch]              gh/guangyey/176/base        -> origin/gh/guangyey/176/base
2025-09-07T07:36:03.9171766Z  * [new branch]              gh/guangyey/176/head        -> origin/gh/guangyey/176/head
2025-09-07T07:36:03.9172968Z  * [new branch]              gh/guangyey/176/orig        -> origin/gh/guangyey/176/orig
2025-09-07T07:36:03.9174636Z  * [new branch]              gh/guangyey/178/base        -> origin/gh/guangyey/178/base
2025-09-07T07:36:03.9175820Z  * [new branch]              gh/guangyey/178/head        -> origin/gh/guangyey/178/head
2025-09-07T07:36:03.9177002Z  * [new branch]              gh/guangyey/178/orig        -> origin/gh/guangyey/178/orig
2025-09-07T07:36:03.9179214Z  * [new branch]              gh/guangyey/181/base        -> origin/gh/guangyey/181/base
2025-09-07T07:36:03.9180432Z  * [new branch]              gh/guangyey/181/head        -> origin/gh/guangyey/181/head
2025-09-07T07:36:03.9181597Z  * [new branch]              gh/guangyey/181/orig        -> origin/gh/guangyey/181/orig
2025-09-07T07:36:03.9183393Z  * [new branch]              gh/guangyey/182/base        -> origin/gh/guangyey/182/base
2025-09-07T07:36:03.9184560Z  * [new branch]              gh/guangyey/182/head        -> origin/gh/guangyey/182/head
2025-09-07T07:36:03.9185744Z  * [new branch]              gh/guangyey/182/orig        -> origin/gh/guangyey/182/orig
2025-09-07T07:36:03.9187490Z  * [new branch]              gh/guangyey/183/base        -> origin/gh/guangyey/183/base
2025-09-07T07:36:03.9188699Z  * [new branch]              gh/guangyey/183/head        -> origin/gh/guangyey/183/head
2025-09-07T07:36:03.9189940Z  * [new branch]              gh/guangyey/183/orig        -> origin/gh/guangyey/183/orig
2025-09-07T07:36:03.9191741Z  * [new branch]              gh/guangyey/184/base        -> origin/gh/guangyey/184/base
2025-09-07T07:36:03.9192998Z  * [new branch]              gh/guangyey/184/head        -> origin/gh/guangyey/184/head
2025-09-07T07:36:03.9194141Z  * [new branch]              gh/guangyey/184/orig        -> origin/gh/guangyey/184/orig
2025-09-07T07:36:03.9195845Z  * [new branch]              gh/guangyey/185/base        -> origin/gh/guangyey/185/base
2025-09-07T07:36:03.9197034Z  * [new branch]              gh/guangyey/185/head        -> origin/gh/guangyey/185/head
2025-09-07T07:36:03.9198245Z  * [new branch]              gh/guangyey/185/orig        -> origin/gh/guangyey/185/orig
2025-09-07T07:36:03.9200057Z  * [new branch]              gh/guangyey/186/base        -> origin/gh/guangyey/186/base
2025-09-07T07:36:03.9201327Z  * [new branch]              gh/guangyey/186/head        -> origin/gh/guangyey/186/head
2025-09-07T07:36:03.9202421Z  * [new branch]              gh/guangyey/186/orig        -> origin/gh/guangyey/186/orig
2025-09-07T07:36:03.9204399Z  * [new branch]              gh/guangyey/187/base        -> origin/gh/guangyey/187/base
2025-09-07T07:36:03.9205635Z  * [new branch]              gh/guangyey/187/head        -> origin/gh/guangyey/187/head
2025-09-07T07:36:03.9207022Z  * [new branch]              gh/guangyey/187/orig        -> origin/gh/guangyey/187/orig
2025-09-07T07:36:03.9208771Z  * [new branch]              gh/guangyey/188/base        -> origin/gh/guangyey/188/base
2025-09-07T07:36:03.9209992Z  * [new branch]              gh/guangyey/188/head        -> origin/gh/guangyey/188/head
2025-09-07T07:36:03.9211171Z  * [new branch]              gh/guangyey/188/orig        -> origin/gh/guangyey/188/orig
2025-09-07T07:36:03.9212928Z  * [new branch]              gh/guangyey/189/base        -> origin/gh/guangyey/189/base
2025-09-07T07:36:03.9214155Z  * [new branch]              gh/guangyey/189/head        -> origin/gh/guangyey/189/head
2025-09-07T07:36:03.9215347Z  * [new branch]              gh/guangyey/189/orig        -> origin/gh/guangyey/189/orig
2025-09-07T07:36:03.9217134Z  * [new branch]              gh/guangyey/190/base        -> origin/gh/guangyey/190/base
2025-09-07T07:36:03.9218447Z  * [new branch]              gh/guangyey/190/head        -> origin/gh/guangyey/190/head
2025-09-07T07:36:03.9219668Z  * [new branch]              gh/guangyey/190/orig        -> origin/gh/guangyey/190/orig
2025-09-07T07:36:03.9221404Z  * [new branch]              gh/guangyey/191/base        -> origin/gh/guangyey/191/base
2025-09-07T07:36:03.9222639Z  * [new branch]              gh/guangyey/191/head        -> origin/gh/guangyey/191/head
2025-09-07T07:36:03.9223810Z  * [new branch]              gh/guangyey/191/orig        -> origin/gh/guangyey/191/orig
2025-09-07T07:36:03.9225575Z  * [new branch]              gh/guangyey/192/base        -> origin/gh/guangyey/192/base
2025-09-07T07:36:03.9226752Z  * [new branch]              gh/guangyey/192/head        -> origin/gh/guangyey/192/head
2025-09-07T07:36:03.9228021Z  * [new branch]              gh/guangyey/192/orig        -> origin/gh/guangyey/192/orig
2025-09-07T07:36:03.9229835Z  * [new branch]              gh/guangyey/193/base        -> origin/gh/guangyey/193/base
2025-09-07T07:36:03.9231015Z  * [new branch]              gh/guangyey/193/head        -> origin/gh/guangyey/193/head
2025-09-07T07:36:03.9232195Z  * [new branch]              gh/guangyey/193/orig        -> origin/gh/guangyey/193/orig
2025-09-07T07:36:03.9234461Z  * [new branch]              gh/guangyey/194/base        -> origin/gh/guangyey/194/base
2025-09-07T07:36:03.9235662Z  * [new branch]              gh/guangyey/194/head        -> origin/gh/guangyey/194/head
2025-09-07T07:36:03.9236851Z  * [new branch]              gh/guangyey/194/orig        -> origin/gh/guangyey/194/orig
2025-09-07T07:36:03.9238587Z  * [new branch]              gh/guangyey/195/base        -> origin/gh/guangyey/195/base
2025-09-07T07:36:03.9239841Z  * [new branch]              gh/guangyey/195/head        -> origin/gh/guangyey/195/head
2025-09-07T07:36:03.9241012Z  * [new branch]              gh/guangyey/195/orig        -> origin/gh/guangyey/195/orig
2025-09-07T07:36:03.9243021Z  * [new branch]              gh/guangyey/196/base        -> origin/gh/guangyey/196/base
2025-09-07T07:36:03.9244137Z  * [new branch]              gh/guangyey/196/head        -> origin/gh/guangyey/196/head
2025-09-07T07:36:03.9245280Z  * [new branch]              gh/guangyey/196/orig        -> origin/gh/guangyey/196/orig
2025-09-07T07:36:03.9246975Z  * [new branch]              gh/guangyey/197/base        -> origin/gh/guangyey/197/base
2025-09-07T07:36:03.9248179Z  * [new branch]              gh/guangyey/197/head        -> origin/gh/guangyey/197/head
2025-09-07T07:36:03.9249394Z  * [new branch]              gh/guangyey/197/orig        -> origin/gh/guangyey/197/orig
2025-09-07T07:36:03.9251223Z  * [new branch]              gh/guangyey/198/base        -> origin/gh/guangyey/198/base
2025-09-07T07:36:03.9252651Z  * [new branch]              gh/guangyey/198/head        -> origin/gh/guangyey/198/head
2025-09-07T07:36:03.9253644Z  * [new branch]              gh/guangyey/198/orig        -> origin/gh/guangyey/198/orig
2025-09-07T07:36:03.9255404Z  * [new branch]              gh/guangyey/199/base        -> origin/gh/guangyey/199/base
2025-09-07T07:36:03.9257146Z  * [new branch]              gh/guangyey/199/head        -> origin/gh/guangyey/199/head
2025-09-07T07:36:03.9258469Z  * [new branch]              gh/guangyey/199/orig        -> origin/gh/guangyey/199/orig
2025-09-07T07:36:03.9260235Z  * [new branch]              gh/guangyey/200/base        -> origin/gh/guangyey/200/base
2025-09-07T07:36:03.9261383Z  * [new branch]              gh/guangyey/200/head        -> origin/gh/guangyey/200/head
2025-09-07T07:36:03.9262733Z  * [new branch]              gh/guangyey/200/orig        -> origin/gh/guangyey/200/orig
2025-09-07T07:36:03.9264274Z  * [new branch]              gh/guangyey/201/base        -> origin/gh/guangyey/201/base
2025-09-07T07:36:03.9265475Z  * [new branch]              gh/guangyey/201/head        -> origin/gh/guangyey/201/head
2025-09-07T07:36:03.9266662Z  * [new branch]              gh/guangyey/201/orig        -> origin/gh/guangyey/201/orig
2025-09-07T07:36:03.9268414Z  * [new branch]              gh/guangyey/202/base        -> origin/gh/guangyey/202/base
2025-09-07T07:36:03.9269755Z  * [new branch]              gh/guangyey/202/head        -> origin/gh/guangyey/202/head
2025-09-07T07:36:03.9270921Z  * [new branch]              gh/guangyey/202/orig        -> origin/gh/guangyey/202/orig
2025-09-07T07:36:03.9272651Z  * [new branch]              gh/guangyey/203/base        -> origin/gh/guangyey/203/base
2025-09-07T07:36:03.9273840Z  * [new branch]              gh/guangyey/203/head        -> origin/gh/guangyey/203/head
2025-09-07T07:36:03.9275013Z  * [new branch]              gh/guangyey/203/orig        -> origin/gh/guangyey/203/orig
2025-09-07T07:36:03.9276735Z  * [new branch]              gh/guangyey/204/base        -> origin/gh/guangyey/204/base
2025-09-07T07:36:03.9278035Z  * [new branch]              gh/guangyey/204/head        -> origin/gh/guangyey/204/head
2025-09-07T07:36:03.9279217Z  * [new branch]              gh/guangyey/204/orig        -> origin/gh/guangyey/204/orig
2025-09-07T07:36:03.9280917Z  * [new branch]              gh/guangyey/205/base        -> origin/gh/guangyey/205/base
2025-09-07T07:36:03.9282105Z  * [new branch]              gh/guangyey/205/head        -> origin/gh/guangyey/205/head
2025-09-07T07:36:03.9283251Z  * [new branch]              gh/guangyey/205/orig        -> origin/gh/guangyey/205/orig
2025-09-07T07:36:03.9285049Z  * [new branch]              gh/guangyey/206/base        -> origin/gh/guangyey/206/base
2025-09-07T07:36:03.9286294Z  * [new branch]              gh/guangyey/206/head        -> origin/gh/guangyey/206/head
2025-09-07T07:36:03.9287471Z  * [new branch]              gh/guangyey/206/orig        -> origin/gh/guangyey/206/orig
2025-09-07T07:36:03.9289222Z  * [new branch]              gh/guangyey/207/base        -> origin/gh/guangyey/207/base
2025-09-07T07:36:03.9290427Z  * [new branch]              gh/guangyey/207/head        -> origin/gh/guangyey/207/head
2025-09-07T07:36:03.9291661Z  * [new branch]              gh/guangyey/207/orig        -> origin/gh/guangyey/207/orig
2025-09-07T07:36:03.9293302Z  * [new branch]              gh/guangyey/79/base         -> origin/gh/guangyey/79/base
2025-09-07T07:36:03.9294494Z  * [new branch]              gh/guangyey/79/head         -> origin/gh/guangyey/79/head
2025-09-07T07:36:03.9295650Z  * [new branch]              gh/guangyey/79/orig         -> origin/gh/guangyey/79/orig
2025-09-07T07:36:03.9297428Z  * [new branch]              gh/guangyey/89/base         -> origin/gh/guangyey/89/base
2025-09-07T07:36:03.9298712Z  * [new branch]              gh/guangyey/89/head         -> origin/gh/guangyey/89/head
2025-09-07T07:36:03.9299977Z  * [new branch]              gh/guangyey/89/orig         -> origin/gh/guangyey/89/orig
2025-09-07T07:36:03.9302289Z  * [new branch]              gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base
2025-09-07T07:36:03.9303675Z  * [new branch]              gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head
2025-09-07T07:36:03.9304927Z  * [new branch]              gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig
2025-09-07T07:36:03.9306752Z  * [new branch]              gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base
2025-09-07T07:36:03.9307777Z  * [new branch]              gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head
2025-09-07T07:36:03.9308948Z  * [new branch]              gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig
2025-09-07T07:36:03.9310685Z  * [new branch]              gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base
2025-09-07T07:36:03.9312036Z  * [new branch]              gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head
2025-09-07T07:36:03.9313419Z  * [new branch]              gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig
2025-09-07T07:36:03.9315128Z  * [new branch]              gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base
2025-09-07T07:36:03.9316307Z  * [new branch]              gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head
2025-09-07T07:36:03.9317509Z  * [new branch]              gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig
2025-09-07T07:36:03.9319348Z  * [new branch]              gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base
2025-09-07T07:36:03.9320956Z  * [new branch]              gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head
2025-09-07T07:36:03.9322166Z  * [new branch]              gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig
2025-09-07T07:36:03.9323909Z  * [new branch]              gh/guilhermeleobas/163/base -> origin/gh/guilhermeleobas/163/base
2025-09-07T07:36:03.9325139Z  * [new branch]              gh/guilhermeleobas/163/head -> origin/gh/guilhermeleobas/163/head
2025-09-07T07:36:03.9326326Z  * [new branch]              gh/guilhermeleobas/163/orig -> origin/gh/guilhermeleobas/163/orig
2025-09-07T07:36:03.9328345Z  * [new branch]              gh/guilhermeleobas/164/base -> origin/gh/guilhermeleobas/164/base
2025-09-07T07:36:03.9329236Z  * [new branch]              gh/guilhermeleobas/164/head -> origin/gh/guilhermeleobas/164/head
2025-09-07T07:36:03.9330435Z  * [new branch]              gh/guilhermeleobas/164/orig -> origin/gh/guilhermeleobas/164/orig
2025-09-07T07:36:03.9332112Z  * [new branch]              gh/guilhermeleobas/165/base -> origin/gh/guilhermeleobas/165/base
2025-09-07T07:36:03.9333321Z  * [new branch]              gh/guilhermeleobas/165/head -> origin/gh/guilhermeleobas/165/head
2025-09-07T07:36:03.9334540Z  * [new branch]              gh/guilhermeleobas/165/orig -> origin/gh/guilhermeleobas/165/orig
2025-09-07T07:36:03.9336310Z  * [new branch]              gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base
2025-09-07T07:36:03.9337513Z  * [new branch]              gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head
2025-09-07T07:36:03.9339229Z  * [new branch]              gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig
2025-09-07T07:36:03.9341096Z  * [new branch]              gh/guilhermeleobas/167/base -> origin/gh/guilhermeleobas/167/base
2025-09-07T07:36:03.9342186Z  * [new branch]              gh/guilhermeleobas/167/head -> origin/gh/guilhermeleobas/167/head
2025-09-07T07:36:03.9343400Z  * [new branch]              gh/guilhermeleobas/167/orig -> origin/gh/guilhermeleobas/167/orig
2025-09-07T07:36:03.9345085Z  * [new branch]              gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base
2025-09-07T07:36:03.9346273Z  * [new branch]              gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head
2025-09-07T07:36:03.9347451Z  * [new branch]              gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig
2025-09-07T07:36:03.9349274Z  * [new branch]              gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base
2025-09-07T07:36:03.9350490Z  * [new branch]              gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head
2025-09-07T07:36:03.9351715Z  * [new branch]              gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig
2025-09-07T07:36:03.9353481Z  * [new branch]              gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base
2025-09-07T07:36:03.9354763Z  * [new branch]              gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head
2025-09-07T07:36:03.9355975Z  * [new branch]              gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig
2025-09-07T07:36:03.9357668Z  * [new branch]              gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base
2025-09-07T07:36:03.9358870Z  * [new branch]              gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head
2025-09-07T07:36:03.9360054Z  * [new branch]              gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig
2025-09-07T07:36:03.9361736Z  * [new branch]              gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base
2025-09-07T07:36:03.9362894Z  * [new branch]              gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head
2025-09-07T07:36:03.9364081Z  * [new branch]              gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig
2025-09-07T07:36:03.9365753Z  * [new branch]              gh/guilhermeleobas/192/base -> origin/gh/guilhermeleobas/192/base
2025-09-07T07:36:03.9367016Z  * [new branch]              gh/guilhermeleobas/192/head -> origin/gh/guilhermeleobas/192/head
2025-09-07T07:36:03.9368192Z  * [new branch]              gh/guilhermeleobas/192/orig -> origin/gh/guilhermeleobas/192/orig
2025-09-07T07:36:03.9370324Z  * [new branch]              gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base
2025-09-07T07:36:03.9371556Z  * [new branch]              gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head
2025-09-07T07:36:03.9372751Z  * [new branch]              gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig
2025-09-07T07:36:03.9374493Z  * [new branch]              gh/guilhermeleobas/194/base -> origin/gh/guilhermeleobas/194/base
2025-09-07T07:36:03.9375673Z  * [new branch]              gh/guilhermeleobas/194/head -> origin/gh/guilhermeleobas/194/head
2025-09-07T07:36:03.9376907Z  * [new branch]              gh/guilhermeleobas/194/orig -> origin/gh/guilhermeleobas/194/orig
2025-09-07T07:36:03.9378751Z  * [new branch]              gh/guilhermeleobas/203/base -> origin/gh/guilhermeleobas/203/base
2025-09-07T07:36:03.9379967Z  * [new branch]              gh/guilhermeleobas/203/head -> origin/gh/guilhermeleobas/203/head
2025-09-07T07:36:03.9381121Z  * [new branch]              gh/guilhermeleobas/203/orig -> origin/gh/guilhermeleobas/203/orig
2025-09-07T07:36:03.9382840Z  * [new branch]              gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base
2025-09-07T07:36:03.9384190Z  * [new branch]              gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head
2025-09-07T07:36:03.9385357Z  * [new branch]              gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig
2025-09-07T07:36:03.9387676Z  * [new branch]              gh/guilhermeleobas/205/base -> origin/gh/guilhermeleobas/205/base
2025-09-07T07:36:03.9388866Z  * [new branch]              gh/guilhermeleobas/205/head -> origin/gh/guilhermeleobas/205/head
2025-09-07T07:36:03.9390091Z  * [new branch]              gh/guilhermeleobas/205/orig -> origin/gh/guilhermeleobas/205/orig
2025-09-07T07:36:03.9392054Z  * [new branch]              gh/guilhermeleobas/209/base -> origin/gh/guilhermeleobas/209/base
2025-09-07T07:36:03.9393250Z  * [new branch]              gh/guilhermeleobas/209/head -> origin/gh/guilhermeleobas/209/head
2025-09-07T07:36:03.9394440Z  * [new branch]              gh/guilhermeleobas/209/orig -> origin/gh/guilhermeleobas/209/orig
2025-09-07T07:36:03.9396289Z  * [new branch]              gh/guilhermeleobas/210/base -> origin/gh/guilhermeleobas/210/base
2025-09-07T07:36:03.9397470Z  * [new branch]              gh/guilhermeleobas/210/head -> origin/gh/guilhermeleobas/210/head
2025-09-07T07:36:03.9398654Z  * [new branch]              gh/guilhermeleobas/210/orig -> origin/gh/guilhermeleobas/210/orig
2025-09-07T07:36:03.9400418Z  * [new branch]              gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base
2025-09-07T07:36:03.9401651Z  * [new branch]              gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head
2025-09-07T07:36:03.9402830Z  * [new branch]              gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig
2025-09-07T07:36:03.9405014Z  * [new branch]              gh/guilhermeleobas/214/base -> origin/gh/guilhermeleobas/214/base
2025-09-07T07:36:03.9406234Z  * [new branch]              gh/guilhermeleobas/214/head -> origin/gh/guilhermeleobas/214/head
2025-09-07T07:36:03.9407468Z  * [new branch]              gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig
2025-09-07T07:36:03.9409193Z  * [new branch]              gh/guilhermeleobas/215/base -> origin/gh/guilhermeleobas/215/base
2025-09-07T07:36:03.9410401Z  * [new branch]              gh/guilhermeleobas/215/head -> origin/gh/guilhermeleobas/215/head
2025-09-07T07:36:03.9411609Z  * [new branch]              gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig
2025-09-07T07:36:03.9413319Z  * [new branch]              gh/guilhermeleobas/216/base -> origin/gh/guilhermeleobas/216/base
2025-09-07T07:36:03.9414523Z  * [new branch]              gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head
2025-09-07T07:36:03.9415704Z  * [new branch]              gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig
2025-09-07T07:36:03.9417516Z  * [new branch]              gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base
2025-09-07T07:36:03.9418845Z  * [new branch]              gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head
2025-09-07T07:36:03.9420212Z  * [new branch]              gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig
2025-09-07T07:36:03.9422045Z  * [new branch]              gh/guilhermeleobas/219/base -> origin/gh/guilhermeleobas/219/base
2025-09-07T07:36:03.9423240Z  * [new branch]              gh/guilhermeleobas/219/head -> origin/gh/guilhermeleobas/219/head
2025-09-07T07:36:03.9424434Z  * [new branch]              gh/guilhermeleobas/219/orig -> origin/gh/guilhermeleobas/219/orig
2025-09-07T07:36:03.9426150Z  * [new branch]              gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base
2025-09-07T07:36:03.9427337Z  * [new branch]              gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head
2025-09-07T07:36:03.9428497Z  * [new branch]              gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig
2025-09-07T07:36:03.9430257Z  * [new branch]              gh/guilhermeleobas/221/base -> origin/gh/guilhermeleobas/221/base
2025-09-07T07:36:03.9431438Z  * [new branch]              gh/guilhermeleobas/221/head -> origin/gh/guilhermeleobas/221/head
2025-09-07T07:36:03.9432770Z  * [new branch]              gh/guilhermeleobas/221/orig -> origin/gh/guilhermeleobas/221/orig
2025-09-07T07:36:03.9434627Z  * [new branch]              gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base
2025-09-07T07:36:03.9435626Z  * [new branch]              gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head
2025-09-07T07:36:03.9436780Z  * [new branch]              gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig
2025-09-07T07:36:03.9438692Z  * [new branch]              gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base
2025-09-07T07:36:03.9439919Z  * [new branch]              gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head
2025-09-07T07:36:03.9441195Z  * [new branch]              gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig
2025-09-07T07:36:03.9442941Z  * [new branch]              gh/guilhermeleobas/224/base -> origin/gh/guilhermeleobas/224/base
2025-09-07T07:36:03.9444162Z  * [new branch]              gh/guilhermeleobas/224/head -> origin/gh/guilhermeleobas/224/head
2025-09-07T07:36:03.9445334Z  * [new branch]              gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig
2025-09-07T07:36:03.9447101Z  * [new branch]              gh/guilhermeleobas/225/base -> origin/gh/guilhermeleobas/225/base
2025-09-07T07:36:03.9448239Z  * [new branch]              gh/guilhermeleobas/225/head -> origin/gh/guilhermeleobas/225/head
2025-09-07T07:36:03.9449428Z  * [new branch]              gh/guilhermeleobas/225/orig -> origin/gh/guilhermeleobas/225/orig
2025-09-07T07:36:03.9451537Z  * [new branch]              gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base
2025-09-07T07:36:03.9452731Z  * [new branch]              gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head
2025-09-07T07:36:03.9453939Z  * [new branch]              gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig
2025-09-07T07:36:03.9455787Z  * [new branch]              gh/guilhermeleobas/227/base -> origin/gh/guilhermeleobas/227/base
2025-09-07T07:36:03.9457096Z  * [new branch]              gh/guilhermeleobas/227/head -> origin/gh/guilhermeleobas/227/head
2025-09-07T07:36:03.9458859Z  * [new branch]              gh/guilhermeleobas/227/orig -> origin/gh/guilhermeleobas/227/orig
2025-09-07T07:36:03.9460689Z  * [new branch]              gh/guilhermeleobas/228/base -> origin/gh/guilhermeleobas/228/base
2025-09-07T07:36:03.9461896Z  * [new branch]              gh/guilhermeleobas/228/head -> origin/gh/guilhermeleobas/228/head
2025-09-07T07:36:03.9463529Z  * [new branch]              gh/guilhermeleobas/228/orig -> origin/gh/guilhermeleobas/228/orig
2025-09-07T07:36:03.9465354Z  * [new branch]              gh/guilhermeleobas/229/base -> origin/gh/guilhermeleobas/229/base
2025-09-07T07:36:03.9466524Z  * [new branch]              gh/guilhermeleobas/229/head -> origin/gh/guilhermeleobas/229/head
2025-09-07T07:36:03.9467747Z  * [new branch]              gh/guilhermeleobas/229/orig -> origin/gh/guilhermeleobas/229/orig
2025-09-07T07:36:03.9469503Z  * [new branch]              gh/guilhermeleobas/230/base -> origin/gh/guilhermeleobas/230/base
2025-09-07T07:36:03.9470697Z  * [new branch]              gh/guilhermeleobas/230/head -> origin/gh/guilhermeleobas/230/head
2025-09-07T07:36:03.9471867Z  * [new branch]              gh/guilhermeleobas/230/orig -> origin/gh/guilhermeleobas/230/orig
2025-09-07T07:36:03.9473672Z  * [new branch]              gh/guilhermeleobas/231/base -> origin/gh/guilhermeleobas/231/base
2025-09-07T07:36:03.9474851Z  * [new branch]              gh/guilhermeleobas/231/head -> origin/gh/guilhermeleobas/231/head
2025-09-07T07:36:03.9476064Z  * [new branch]              gh/guilhermeleobas/231/orig -> origin/gh/guilhermeleobas/231/orig
2025-09-07T07:36:03.9477805Z  * [new branch]              gh/guilhermeleobas/232/base -> origin/gh/guilhermeleobas/232/base
2025-09-07T07:36:03.9479002Z  * [new branch]              gh/guilhermeleobas/232/head -> origin/gh/guilhermeleobas/232/head
2025-09-07T07:36:03.9480275Z  * [new branch]              gh/guilhermeleobas/232/orig -> origin/gh/guilhermeleobas/232/orig
2025-09-07T07:36:03.9482092Z  * [new branch]              gh/guilhermeleobas/233/base -> origin/gh/guilhermeleobas/233/base
2025-09-07T07:36:03.9483160Z  * [new branch]              gh/guilhermeleobas/233/head -> origin/gh/guilhermeleobas/233/head
2025-09-07T07:36:03.9484362Z  * [new branch]              gh/guilhermeleobas/233/orig -> origin/gh/guilhermeleobas/233/orig
2025-09-07T07:36:03.9486135Z  * [new branch]              gh/guilhermeleobas/234/base -> origin/gh/guilhermeleobas/234/base
2025-09-07T07:36:03.9487365Z  * [new branch]              gh/guilhermeleobas/234/head -> origin/gh/guilhermeleobas/234/head
2025-09-07T07:36:03.9488939Z  * [new branch]              gh/guilhermeleobas/234/orig -> origin/gh/guilhermeleobas/234/orig
2025-09-07T07:36:03.9490737Z  * [new branch]              gh/guilhermeleobas/235/base -> origin/gh/guilhermeleobas/235/base
2025-09-07T07:36:03.9491927Z  * [new branch]              gh/guilhermeleobas/235/head -> origin/gh/guilhermeleobas/235/head
2025-09-07T07:36:03.9493300Z  * [new branch]              gh/guilhermeleobas/235/orig -> origin/gh/guilhermeleobas/235/orig
2025-09-07T07:36:03.9495053Z  * [new branch]              gh/guilhermeleobas/236/base -> origin/gh/guilhermeleobas/236/base
2025-09-07T07:36:03.9496205Z  * [new branch]              gh/guilhermeleobas/236/head -> origin/gh/guilhermeleobas/236/head
2025-09-07T07:36:03.9497407Z  * [new branch]              gh/guilhermeleobas/236/orig -> origin/gh/guilhermeleobas/236/orig
2025-09-07T07:36:03.9499297Z  * [new branch]              gh/guilhermeleobas/237/base -> origin/gh/guilhermeleobas/237/base
2025-09-07T07:36:03.9500484Z  * [new branch]              gh/guilhermeleobas/237/head -> origin/gh/guilhermeleobas/237/head
2025-09-07T07:36:03.9501664Z  * [new branch]              gh/guilhermeleobas/237/orig -> origin/gh/guilhermeleobas/237/orig
2025-09-07T07:36:03.9503588Z  * [new branch]              gh/guilhermeleobas/238/base -> origin/gh/guilhermeleobas/238/base
2025-09-07T07:36:03.9504850Z  * [new branch]              gh/guilhermeleobas/238/head -> origin/gh/guilhermeleobas/238/head
2025-09-07T07:36:03.9506029Z  * [new branch]              gh/guilhermeleobas/238/orig -> origin/gh/guilhermeleobas/238/orig
2025-09-07T07:36:03.9507850Z  * [new branch]              gh/guilhermeleobas/239/base -> origin/gh/guilhermeleobas/239/base
2025-09-07T07:36:03.9509018Z  * [new branch]              gh/guilhermeleobas/239/head -> origin/gh/guilhermeleobas/239/head
2025-09-07T07:36:03.9510292Z  * [new branch]              gh/guilhermeleobas/239/orig -> origin/gh/guilhermeleobas/239/orig
2025-09-07T07:36:03.9512128Z  * [new branch]              gh/guilhermeleobas/240/base -> origin/gh/guilhermeleobas/240/base
2025-09-07T07:36:03.9513349Z  * [new branch]              gh/guilhermeleobas/240/head -> origin/gh/guilhermeleobas/240/head
2025-09-07T07:36:03.9514550Z  * [new branch]              gh/guilhermeleobas/240/orig -> origin/gh/guilhermeleobas/240/orig
2025-09-07T07:36:03.9516337Z  * [new branch]              gh/guilhermeleobas/241/base -> origin/gh/guilhermeleobas/241/base
2025-09-07T07:36:03.9517602Z  * [new branch]              gh/guilhermeleobas/241/head -> origin/gh/guilhermeleobas/241/head
2025-09-07T07:36:03.9518790Z  * [new branch]              gh/guilhermeleobas/241/orig -> origin/gh/guilhermeleobas/241/orig
2025-09-07T07:36:03.9520547Z  * [new branch]              gh/guilhermeleobas/242/base -> origin/gh/guilhermeleobas/242/base
2025-09-07T07:36:03.9521809Z  * [new branch]              gh/guilhermeleobas/242/head -> origin/gh/guilhermeleobas/242/head
2025-09-07T07:36:03.9522979Z  * [new branch]              gh/guilhermeleobas/242/orig -> origin/gh/guilhermeleobas/242/orig
2025-09-07T07:36:03.9524703Z  * [new branch]              gh/guilhermeleobas/243/base -> origin/gh/guilhermeleobas/243/base
2025-09-07T07:36:03.9525932Z  * [new branch]              gh/guilhermeleobas/243/head -> origin/gh/guilhermeleobas/243/head
2025-09-07T07:36:03.9527117Z  * [new branch]              gh/guilhermeleobas/243/orig -> origin/gh/guilhermeleobas/243/orig
2025-09-07T07:36:03.9529167Z  * [new branch]              gh/guilhermeleobas/244/base -> origin/gh/guilhermeleobas/244/base
2025-09-07T07:36:03.9530241Z  * [new branch]              gh/guilhermeleobas/244/head -> origin/gh/guilhermeleobas/244/head
2025-09-07T07:36:03.9531415Z  * [new branch]              gh/guilhermeleobas/244/orig -> origin/gh/guilhermeleobas/244/orig
2025-09-07T07:36:03.9533129Z  * [new branch]              gh/guilhermeleobas/245/base -> origin/gh/guilhermeleobas/245/base
2025-09-07T07:36:03.9534340Z  * [new branch]              gh/guilhermeleobas/245/head -> origin/gh/guilhermeleobas/245/head
2025-09-07T07:36:03.9535513Z  * [new branch]              gh/guilhermeleobas/245/orig -> origin/gh/guilhermeleobas/245/orig
2025-09-07T07:36:03.9537281Z  * [new branch]              gh/guilhermeleobas/73/base  -> origin/gh/guilhermeleobas/73/base
2025-09-07T07:36:03.9538596Z  * [new branch]              gh/guilhermeleobas/73/head  -> origin/gh/guilhermeleobas/73/head
2025-09-07T07:36:03.9539800Z  * [new branch]              gh/guilhermeleobas/73/orig  -> origin/gh/guilhermeleobas/73/orig
2025-09-07T07:36:03.9541992Z  * [new branch]              gh/henrylhtsang/140/base    -> origin/gh/henrylhtsang/140/base
2025-09-07T07:36:03.9543230Z  * [new branch]              gh/henrylhtsang/140/head    -> origin/gh/henrylhtsang/140/head
2025-09-07T07:36:03.9544434Z  * [new branch]              gh/henrylhtsang/140/orig    -> origin/gh/henrylhtsang/140/orig
2025-09-07T07:36:03.9546068Z  * [new branch]              gh/henrylhtsang/141/base    -> origin/gh/henrylhtsang/141/base
2025-09-07T07:36:03.9547273Z  * [new branch]              gh/henrylhtsang/141/head    -> origin/gh/henrylhtsang/141/head
2025-09-07T07:36:03.9548524Z  * [new branch]              gh/henrylhtsang/141/orig    -> origin/gh/henrylhtsang/141/orig
2025-09-07T07:36:03.9550432Z  * [new branch]              gh/henrylhtsang/142/base    -> origin/gh/henrylhtsang/142/base
2025-09-07T07:36:03.9551731Z  * [new branch]              gh/henrylhtsang/142/head    -> origin/gh/henrylhtsang/142/head
2025-09-07T07:36:03.9552937Z  * [new branch]              gh/henrylhtsang/142/orig    -> origin/gh/henrylhtsang/142/orig
2025-09-07T07:36:03.9554620Z  * [new branch]              gh/henrylhtsang/143/base    -> origin/gh/henrylhtsang/143/base
2025-09-07T07:36:03.9555812Z  * [new branch]              gh/henrylhtsang/143/head    -> origin/gh/henrylhtsang/143/head
2025-09-07T07:36:03.9557035Z  * [new branch]              gh/henrylhtsang/143/orig    -> origin/gh/henrylhtsang/143/orig
2025-09-07T07:36:03.9558791Z  * [new branch]              gh/henrylhtsang/144/base    -> origin/gh/henrylhtsang/144/base
2025-09-07T07:36:03.9559974Z  * [new branch]              gh/henrylhtsang/144/head    -> origin/gh/henrylhtsang/144/head
2025-09-07T07:36:03.9561160Z  * [new branch]              gh/henrylhtsang/144/orig    -> origin/gh/henrylhtsang/144/orig
2025-09-07T07:36:03.9563005Z  * [new branch]              gh/henrylhtsang/145/base    -> origin/gh/henrylhtsang/145/base
2025-09-07T07:36:03.9564314Z  * [new branch]              gh/henrylhtsang/145/head    -> origin/gh/henrylhtsang/145/head
2025-09-07T07:36:03.9565493Z  * [new branch]              gh/henrylhtsang/145/orig    -> origin/gh/henrylhtsang/145/orig
2025-09-07T07:36:03.9567265Z  * [new branch]              gh/henrylhtsang/146/base    -> origin/gh/henrylhtsang/146/base
2025-09-07T07:36:03.9568493Z  * [new branch]              gh/henrylhtsang/146/head    -> origin/gh/henrylhtsang/146/head
2025-09-07T07:36:03.9569680Z  * [new branch]              gh/henrylhtsang/146/orig    -> origin/gh/henrylhtsang/146/orig
2025-09-07T07:36:03.9571405Z  * [new branch]              gh/henrylhtsang/147/base    -> origin/gh/henrylhtsang/147/base
2025-09-07T07:36:03.9572597Z  * [new branch]              gh/henrylhtsang/147/head    -> origin/gh/henrylhtsang/147/head
2025-09-07T07:36:03.9573816Z  * [new branch]              gh/henrylhtsang/147/orig    -> origin/gh/henrylhtsang/147/orig
2025-09-07T07:36:03.9575689Z  * [new branch]              gh/henrylhtsang/148/base    -> origin/gh/henrylhtsang/148/base
2025-09-07T07:36:03.9577593Z  * [new branch]              gh/henrylhtsang/148/head    -> origin/gh/henrylhtsang/148/head
2025-09-07T07:36:03.9578955Z  * [new branch]              gh/henrylhtsang/148/orig    -> origin/gh/henrylhtsang/148/orig
2025-09-07T07:36:03.9580624Z  * [new branch]              gh/henrylhtsang/149/base    -> origin/gh/henrylhtsang/149/base
2025-09-07T07:36:03.9581963Z  * [new branch]              gh/henrylhtsang/149/head    -> origin/gh/henrylhtsang/149/head
2025-09-07T07:36:03.9583243Z  * [new branch]              gh/henrylhtsang/149/orig    -> origin/gh/henrylhtsang/149/orig
2025-09-07T07:36:03.9585582Z  * [new branch]              gh/huydhn/1/next            -> origin/gh/huydhn/1/next
2025-09-07T07:36:03.9587209Z  * [new branch]              gh/huydhn/2/next            -> origin/gh/huydhn/2/next
2025-09-07T07:36:03.9588890Z  * [new branch]              gh/huydhn/3/next            -> origin/gh/huydhn/3/next
2025-09-07T07:36:03.9590699Z  * [new branch]              gh/huydhn/4/next            -> origin/gh/huydhn/4/next
2025-09-07T07:36:03.9592352Z  * [new branch]              gh/huydhn/5/next            -> origin/gh/huydhn/5/next
2025-09-07T07:36:03.9594081Z  * [new branch]              gh/huydhn/6/next            -> origin/gh/huydhn/6/next
2025-09-07T07:36:03.9596278Z  * [new branch]              gh/int3/97/base             -> origin/gh/int3/97/base
2025-09-07T07:36:03.9597473Z  * [new branch]              gh/int3/97/head             -> origin/gh/int3/97/head
2025-09-07T07:36:03.9600152Z  * [new branch]              gh/isuruf/101/base          -> origin/gh/isuruf/101/base
2025-09-07T07:36:03.9601352Z  * [new branch]              gh/isuruf/101/head          -> origin/gh/isuruf/101/head
2025-09-07T07:36:03.9603028Z  * [new branch]              gh/isuruf/141/base          -> origin/gh/isuruf/141/base
2025-09-07T07:36:03.9604630Z  * [new branch]              gh/isuruf/141/head          -> origin/gh/isuruf/141/head
2025-09-07T07:36:03.9605830Z  * [new branch]              gh/isuruf/141/orig          -> origin/gh/isuruf/141/orig
2025-09-07T07:36:03.9607652Z  * [new branch]              gh/isuruf/142/base          -> origin/gh/isuruf/142/base
2025-09-07T07:36:03.9608835Z  * [new branch]              gh/isuruf/142/head          -> origin/gh/isuruf/142/head
2025-09-07T07:36:03.9610045Z  * [new branch]              gh/isuruf/142/orig          -> origin/gh/isuruf/142/orig
2025-09-07T07:36:03.9611701Z  * [new branch]              gh/isuruf/143/base          -> origin/gh/isuruf/143/base
2025-09-07T07:36:03.9612858Z  * [new branch]              gh/isuruf/143/head          -> origin/gh/isuruf/143/head
2025-09-07T07:36:03.9614037Z  * [new branch]              gh/isuruf/143/orig          -> origin/gh/isuruf/143/orig
2025-09-07T07:36:03.9615692Z  * [new branch]              gh/isuruf/144/base          -> origin/gh/isuruf/144/base
2025-09-07T07:36:03.9616853Z  * [new branch]              gh/isuruf/144/head          -> origin/gh/isuruf/144/head
2025-09-07T07:36:03.9618103Z  * [new branch]              gh/isuruf/144/orig          -> origin/gh/isuruf/144/orig
2025-09-07T07:36:03.9619805Z  * [new branch]              gh/isuruf/145/base          -> origin/gh/isuruf/145/base
2025-09-07T07:36:03.9621060Z  * [new branch]              gh/isuruf/145/head          -> origin/gh/isuruf/145/head
2025-09-07T07:36:03.9622311Z  * [new branch]              gh/isuruf/145/orig          -> origin/gh/isuruf/145/orig
2025-09-07T07:36:03.9623992Z  * [new branch]              gh/isuruf/146/base          -> origin/gh/isuruf/146/base
2025-09-07T07:36:03.9625195Z  * [new branch]              gh/isuruf/146/head          -> origin/gh/isuruf/146/head
2025-09-07T07:36:03.9626401Z  * [new branch]              gh/isuruf/146/orig          -> origin/gh/isuruf/146/orig
2025-09-07T07:36:03.9628118Z  * [new branch]              gh/isuruf/81/base           -> origin/gh/isuruf/81/base
2025-09-07T07:36:03.9629326Z  * [new branch]              gh/isuruf/81/head           -> origin/gh/isuruf/81/head
2025-09-07T07:36:03.9630581Z  * [new branch]              gh/isuruf/81/orig           -> origin/gh/isuruf/81/orig
2025-09-07T07:36:03.9632620Z  * [new branch]              gh/jamesjwu/150/base        -> origin/gh/jamesjwu/150/base
2025-09-07T07:36:03.9633761Z  * [new branch]              gh/jamesjwu/150/head        -> origin/gh/jamesjwu/150/head
2025-09-07T07:36:03.9634929Z  * [new branch]              gh/jamesjwu/150/orig        -> origin/gh/jamesjwu/150/orig
2025-09-07T07:36:03.9636818Z  * [new branch]              gh/jamesjwu/154/base        -> origin/gh/jamesjwu/154/base
2025-09-07T07:36:03.9637955Z  * [new branch]              gh/jamesjwu/154/head        -> origin/gh/jamesjwu/154/head
2025-09-07T07:36:03.9639251Z  * [new branch]              gh/jamesjwu/154/orig        -> origin/gh/jamesjwu/154/orig
2025-09-07T07:36:03.9640962Z  * [new branch]              gh/jamesjwu/155/base        -> origin/gh/jamesjwu/155/base
2025-09-07T07:36:03.9642138Z  * [new branch]              gh/jamesjwu/155/head        -> origin/gh/jamesjwu/155/head
2025-09-07T07:36:03.9643331Z  * [new branch]              gh/jamesjwu/155/orig        -> origin/gh/jamesjwu/155/orig
2025-09-07T07:36:03.9645030Z  * [new branch]              gh/jamesjwu/159/base        -> origin/gh/jamesjwu/159/base
2025-09-07T07:36:03.9646213Z  * [new branch]              gh/jamesjwu/159/head        -> origin/gh/jamesjwu/159/head
2025-09-07T07:36:03.9647513Z  * [new branch]              gh/jamesjwu/159/orig        -> origin/gh/jamesjwu/159/orig
2025-09-07T07:36:03.9649457Z  * [new branch]              gh/jamesjwu/163/base        -> origin/gh/jamesjwu/163/base
2025-09-07T07:36:03.9650672Z  * [new branch]              gh/jamesjwu/163/head        -> origin/gh/jamesjwu/163/head
2025-09-07T07:36:03.9651831Z  * [new branch]              gh/jamesjwu/163/orig        -> origin/gh/jamesjwu/163/orig
2025-09-07T07:36:03.9654057Z  * [new branch]              gh/jamesjwu/171/base        -> origin/gh/jamesjwu/171/base
2025-09-07T07:36:03.9655274Z  * [new branch]              gh/jamesjwu/171/head        -> origin/gh/jamesjwu/171/head
2025-09-07T07:36:03.9656422Z  * [new branch]              gh/jamesjwu/171/orig        -> origin/gh/jamesjwu/171/orig
2025-09-07T07:36:03.9658200Z  * [new branch]              gh/jamesjwu/176/base        -> origin/gh/jamesjwu/176/base
2025-09-07T07:36:03.9659418Z  * [new branch]              gh/jamesjwu/176/head        -> origin/gh/jamesjwu/176/head
2025-09-07T07:36:03.9660631Z  * [new branch]              gh/jamesjwu/176/orig        -> origin/gh/jamesjwu/176/orig
2025-09-07T07:36:03.9662364Z  * [new branch]              gh/jamesjwu/181/base        -> origin/gh/jamesjwu/181/base
2025-09-07T07:36:03.9663537Z  * [new branch]              gh/jamesjwu/181/head        -> origin/gh/jamesjwu/181/head
2025-09-07T07:36:03.9664704Z  * [new branch]              gh/jamesjwu/181/orig        -> origin/gh/jamesjwu/181/orig
2025-09-07T07:36:03.9666806Z  * [new branch]              gh/jamesjwu/182/base        -> origin/gh/jamesjwu/182/base
2025-09-07T07:36:03.9668020Z  * [new branch]              gh/jamesjwu/182/head        -> origin/gh/jamesjwu/182/head
2025-09-07T07:36:03.9669220Z  * [new branch]              gh/jamesjwu/182/orig        -> origin/gh/jamesjwu/182/orig
2025-09-07T07:36:03.9670994Z  * [new branch]              gh/jamesjwu/183/base        -> origin/gh/jamesjwu/183/base
2025-09-07T07:36:03.9672615Z  * [new branch]              gh/jamesjwu/183/head        -> origin/gh/jamesjwu/183/head
2025-09-07T07:36:03.9673839Z  * [new branch]              gh/jamesjwu/183/orig        -> origin/gh/jamesjwu/183/orig
2025-09-07T07:36:03.9675792Z  * [new branch]              gh/jamesjwu/184/base        -> origin/gh/jamesjwu/184/base
2025-09-07T07:36:03.9676876Z  * [new branch]              gh/jamesjwu/184/head        -> origin/gh/jamesjwu/184/head
2025-09-07T07:36:03.9678082Z  * [new branch]              gh/jamesjwu/184/orig        -> origin/gh/jamesjwu/184/orig
2025-09-07T07:36:03.9679778Z  * [new branch]              gh/jamesjwu/185/base        -> origin/gh/jamesjwu/185/base
2025-09-07T07:36:03.9681056Z  * [new branch]              gh/jamesjwu/185/head        -> origin/gh/jamesjwu/185/head
2025-09-07T07:36:03.9682239Z  * [new branch]              gh/jamesjwu/185/orig        -> origin/gh/jamesjwu/185/orig
2025-09-07T07:36:03.9683911Z  * [new branch]              gh/jamesjwu/186/base        -> origin/gh/jamesjwu/186/base
2025-09-07T07:36:03.9685093Z  * [new branch]              gh/jamesjwu/186/head        -> origin/gh/jamesjwu/186/head
2025-09-07T07:36:03.9686434Z  * [new branch]              gh/jamesjwu/186/orig        -> origin/gh/jamesjwu/186/orig
2025-09-07T07:36:03.9688291Z  * [new branch]              gh/jamesjwu/187/base        -> origin/gh/jamesjwu/187/base
2025-09-07T07:36:03.9689695Z  * [new branch]              gh/jamesjwu/187/head        -> origin/gh/jamesjwu/187/head
2025-09-07T07:36:03.9690874Z  * [new branch]              gh/jamesjwu/187/orig        -> origin/gh/jamesjwu/187/orig
2025-09-07T07:36:03.9692678Z  * [new branch]              gh/jamesjwu/188/base        -> origin/gh/jamesjwu/188/base
2025-09-07T07:36:03.9693937Z  * [new branch]              gh/jamesjwu/188/head        -> origin/gh/jamesjwu/188/head
2025-09-07T07:36:03.9695128Z  * [new branch]              gh/jamesjwu/188/orig        -> origin/gh/jamesjwu/188/orig
2025-09-07T07:36:03.9697404Z  * [new branch]              gh/jamesjwu/189/base        -> origin/gh/jamesjwu/189/base
2025-09-07T07:36:03.9698760Z  * [new branch]              gh/jamesjwu/189/head        -> origin/gh/jamesjwu/189/head
2025-09-07T07:36:03.9699969Z  * [new branch]              gh/jamesjwu/189/orig        -> origin/gh/jamesjwu/189/orig
2025-09-07T07:36:03.9702150Z  * [new branch]              gh/jamesjwu/190/base        -> origin/gh/jamesjwu/190/base
2025-09-07T07:36:03.9703486Z  * [new branch]              gh/jamesjwu/190/head        -> origin/gh/jamesjwu/190/head
2025-09-07T07:36:03.9706997Z  * [new branch]              gh/jamesjwu/190/orig        -> origin/gh/jamesjwu/190/orig
2025-09-07T07:36:03.9708945Z  * [new branch]              gh/jamesjwu/52/base         -> origin/gh/jamesjwu/52/base
2025-09-07T07:36:03.9710137Z  * [new branch]              gh/jamesjwu/52/head         -> origin/gh/jamesjwu/52/head
2025-09-07T07:36:03.9711800Z  * [new branch]              gh/jamesjwu/53/base         -> origin/gh/jamesjwu/53/base
2025-09-07T07:36:03.9712966Z  * [new branch]              gh/jamesjwu/53/head         -> origin/gh/jamesjwu/53/head
2025-09-07T07:36:03.9714625Z  * [new branch]              gh/jamesjwu/54/base         -> origin/gh/jamesjwu/54/base
2025-09-07T07:36:03.9715997Z  * [new branch]              gh/jamesjwu/54/head         -> origin/gh/jamesjwu/54/head
2025-09-07T07:36:03.9717642Z  * [new branch]              gh/jamesjwu/55/base         -> origin/gh/jamesjwu/55/base
2025-09-07T07:36:03.9718790Z  * [new branch]              gh/jamesjwu/55/head         -> origin/gh/jamesjwu/55/head
2025-09-07T07:36:03.9720437Z  * [new branch]              gh/jamesjwu/56/base         -> origin/gh/jamesjwu/56/base
2025-09-07T07:36:03.9721570Z  * [new branch]              gh/jamesjwu/56/head         -> origin/gh/jamesjwu/56/head
2025-09-07T07:36:03.9723207Z  * [new branch]              gh/jamesjwu/57/base         -> origin/gh/jamesjwu/57/base
2025-09-07T07:36:03.9724445Z  * [new branch]              gh/jamesjwu/57/head         -> origin/gh/jamesjwu/57/head
2025-09-07T07:36:03.9726083Z  * [new branch]              gh/jamesjwu/58/base         -> origin/gh/jamesjwu/58/base
2025-09-07T07:36:03.9727266Z  * [new branch]              gh/jamesjwu/58/head         -> origin/gh/jamesjwu/58/head
2025-09-07T07:36:03.9728915Z  * [new branch]              gh/jamesjwu/59/base         -> origin/gh/jamesjwu/59/base
2025-09-07T07:36:03.9730089Z  * [new branch]              gh/jamesjwu/59/head         -> origin/gh/jamesjwu/59/head
2025-09-07T07:36:03.9731687Z  * [new branch]              gh/jamesjwu/60/base         -> origin/gh/jamesjwu/60/base
2025-09-07T07:36:03.9732834Z  * [new branch]              gh/jamesjwu/60/head         -> origin/gh/jamesjwu/60/head
2025-09-07T07:36:03.9734565Z  * [new branch]              gh/jamesjwu/61/base         -> origin/gh/jamesjwu/61/base
2025-09-07T07:36:03.9735805Z  * [new branch]              gh/jamesjwu/61/head         -> origin/gh/jamesjwu/61/head
2025-09-07T07:36:03.9737416Z  * [new branch]              gh/jamesjwu/62/base         -> origin/gh/jamesjwu/62/base
2025-09-07T07:36:03.9738653Z  * [new branch]              gh/jamesjwu/62/head         -> origin/gh/jamesjwu/62/head
2025-09-07T07:36:03.9740352Z  * [new branch]              gh/jamesjwu/63/base         -> origin/gh/jamesjwu/63/base
2025-09-07T07:36:03.9741553Z  * [new branch]              gh/jamesjwu/63/head         -> origin/gh/jamesjwu/63/head
2025-09-07T07:36:03.9743467Z  * [new branch]              gh/jamesjwu/64/base         -> origin/gh/jamesjwu/64/base
2025-09-07T07:36:03.9744657Z  * [new branch]              gh/jamesjwu/64/head         -> origin/gh/jamesjwu/64/head
2025-09-07T07:36:03.9746284Z  * [new branch]              gh/jamesjwu/65/base         -> origin/gh/jamesjwu/65/base
2025-09-07T07:36:03.9747481Z  * [new branch]              gh/jamesjwu/65/head         -> origin/gh/jamesjwu/65/head
2025-09-07T07:36:03.9749614Z  * [new branch]              gh/janeyx99/165/base        -> origin/gh/janeyx99/165/base
2025-09-07T07:36:03.9750839Z  * [new branch]              gh/janeyx99/165/head        -> origin/gh/janeyx99/165/head
2025-09-07T07:36:03.9752146Z  * [new branch]              gh/janeyx99/165/orig        -> origin/gh/janeyx99/165/orig
2025-09-07T07:36:03.9753742Z  * [new branch]              gh/janeyx99/201/base        -> origin/gh/janeyx99/201/base
2025-09-07T07:36:03.9755029Z  * [new branch]              gh/janeyx99/201/head        -> origin/gh/janeyx99/201/head
2025-09-07T07:36:03.9756208Z  * [new branch]              gh/janeyx99/201/orig        -> origin/gh/janeyx99/201/orig
2025-09-07T07:36:03.9758270Z  * [new branch]              gh/janeyx99/225/base        -> origin/gh/janeyx99/225/base
2025-09-07T07:36:03.9759482Z  * [new branch]              gh/janeyx99/225/head        -> origin/gh/janeyx99/225/head
2025-09-07T07:36:03.9760676Z  * [new branch]              gh/janeyx99/225/orig        -> origin/gh/janeyx99/225/orig
2025-09-07T07:36:03.9762391Z  * [new branch]              gh/janeyx99/296/base        -> origin/gh/janeyx99/296/base
2025-09-07T07:36:03.9763574Z  * [new branch]              gh/janeyx99/296/head        -> origin/gh/janeyx99/296/head
2025-09-07T07:36:03.9764786Z  * [new branch]              gh/janeyx99/296/orig        -> origin/gh/janeyx99/296/orig
2025-09-07T07:36:03.9766458Z  * [new branch]              gh/janeyx99/297/base        -> origin/gh/janeyx99/297/base
2025-09-07T07:36:03.9767652Z  * [new branch]              gh/janeyx99/297/head        -> origin/gh/janeyx99/297/head
2025-09-07T07:36:03.9768820Z  * [new branch]              gh/janeyx99/297/orig        -> origin/gh/janeyx99/297/orig
2025-09-07T07:36:03.9770972Z  * [new branch]              gh/janeyx99/298/base        -> origin/gh/janeyx99/298/base
2025-09-07T07:36:03.9772244Z  * [new branch]              gh/janeyx99/298/head        -> origin/gh/janeyx99/298/head
2025-09-07T07:36:03.9773444Z  * [new branch]              gh/janeyx99/298/orig        -> origin/gh/janeyx99/298/orig
2025-09-07T07:36:03.9775197Z  * [new branch]              gh/janeyx99/299/base        -> origin/gh/janeyx99/299/base
2025-09-07T07:36:03.9776430Z  * [new branch]              gh/janeyx99/299/head        -> origin/gh/janeyx99/299/head
2025-09-07T07:36:03.9777734Z  * [new branch]              gh/janeyx99/299/orig        -> origin/gh/janeyx99/299/orig
2025-09-07T07:36:03.9779584Z  * [new branch]              gh/janeyx99/300/base        -> origin/gh/janeyx99/300/base
2025-09-07T07:36:03.9780936Z  * [new branch]              gh/janeyx99/300/head        -> origin/gh/janeyx99/300/head
2025-09-07T07:36:03.9782122Z  * [new branch]              gh/janeyx99/300/orig        -> origin/gh/janeyx99/300/orig
2025-09-07T07:36:03.9784396Z  * [new branch]              gh/janeyx99/301/base        -> origin/gh/janeyx99/301/base
2025-09-07T07:36:03.9785745Z  * [new branch]              gh/janeyx99/301/head        -> origin/gh/janeyx99/301/head
2025-09-07T07:36:03.9786899Z  * [new branch]              gh/janeyx99/301/orig        -> origin/gh/janeyx99/301/orig
2025-09-07T07:36:03.9788675Z  * [new branch]              gh/janeyx99/302/base        -> origin/gh/janeyx99/302/base
2025-09-07T07:36:03.9789981Z  * [new branch]              gh/janeyx99/302/head        -> origin/gh/janeyx99/302/head
2025-09-07T07:36:03.9791785Z  * [new branch]              gh/janeyx99/303/base        -> origin/gh/janeyx99/303/base
2025-09-07T07:36:03.9792948Z  * [new branch]              gh/janeyx99/303/head        -> origin/gh/janeyx99/303/head
2025-09-07T07:36:03.9794941Z  * [new branch]              gh/janeyx99/88/base         -> origin/gh/janeyx99/88/base
2025-09-07T07:36:03.9796159Z  * [new branch]              gh/janeyx99/88/head         -> origin/gh/janeyx99/88/head
2025-09-07T07:36:03.9797328Z  * [new branch]              gh/janeyx99/88/orig         -> origin/gh/janeyx99/88/orig
2025-09-07T07:36:03.9799499Z  * [new branch]              gh/jansel/360/base          -> origin/gh/jansel/360/base
2025-09-07T07:36:03.9800664Z  * [new branch]              gh/jansel/360/head          -> origin/gh/jansel/360/head
2025-09-07T07:36:03.9802383Z  * [new branch]              gh/jansel/451/base          -> origin/gh/jansel/451/base
2025-09-07T07:36:03.9803768Z  * [new branch]              gh/jansel/451/head          -> origin/gh/jansel/451/head
2025-09-07T07:36:03.9805002Z  * [new branch]              gh/jansel/451/orig          -> origin/gh/jansel/451/orig
2025-09-07T07:36:03.9806809Z  * [new branch]              gh/jansel/462/base          -> origin/gh/jansel/462/base
2025-09-07T07:36:03.9807984Z  * [new branch]              gh/jansel/462/head          -> origin/gh/jansel/462/head
2025-09-07T07:36:03.9809138Z  * [new branch]              gh/jansel/462/orig          -> origin/gh/jansel/462/orig
2025-09-07T07:36:03.9810862Z  * [new branch]              gh/jansel/531/base          -> origin/gh/jansel/531/base
2025-09-07T07:36:03.9812049Z  * [new branch]              gh/jansel/531/head          -> origin/gh/jansel/531/head
2025-09-07T07:36:03.9813246Z  * [new branch]              gh/jansel/531/orig          -> origin/gh/jansel/531/orig
2025-09-07T07:36:03.9815427Z  * [new branch]              gh/jbschlosser/208/head     -> origin/gh/jbschlosser/208/head
2025-09-07T07:36:03.9817193Z  * [new branch]              gh/jbschlosser/247/base     -> origin/gh/jbschlosser/247/base
2025-09-07T07:36:03.9818514Z  * [new branch]              gh/jbschlosser/247/head     -> origin/gh/jbschlosser/247/head
2025-09-07T07:36:03.9819701Z  * [new branch]              gh/jbschlosser/247/orig     -> origin/gh/jbschlosser/247/orig
2025-09-07T07:36:03.9821551Z  * [new branch]              gh/jbschlosser/248/base     -> origin/gh/jbschlosser/248/base
2025-09-07T07:36:03.9822760Z  * [new branch]              gh/jbschlosser/248/head     -> origin/gh/jbschlosser/248/head
2025-09-07T07:36:03.9823932Z  * [new branch]              gh/jbschlosser/248/orig     -> origin/gh/jbschlosser/248/orig
2025-09-07T07:36:03.9825901Z  * [new branch]              gh/jbschlosser/250/base     -> origin/gh/jbschlosser/250/base
2025-09-07T07:36:03.9827163Z  * [new branch]              gh/jbschlosser/250/head     -> origin/gh/jbschlosser/250/head
2025-09-07T07:36:03.9828355Z  * [new branch]              gh/jbschlosser/250/orig     -> origin/gh/jbschlosser/250/orig
2025-09-07T07:36:03.9830450Z  * [new branch]              gh/jiayisunx/59/base        -> origin/gh/jiayisunx/59/base
2025-09-07T07:36:03.9831653Z  * [new branch]              gh/jiayisunx/59/head        -> origin/gh/jiayisunx/59/head
2025-09-07T07:36:03.9832850Z  * [new branch]              gh/jiayisunx/59/orig        -> origin/gh/jiayisunx/59/orig
2025-09-07T07:36:03.9834477Z  * [new branch]              gh/jiayisunx/61/base        -> origin/gh/jiayisunx/61/base
2025-09-07T07:36:03.9835676Z  * [new branch]              gh/jiayisunx/61/head        -> origin/gh/jiayisunx/61/head
2025-09-07T07:36:03.9836959Z  * [new branch]              gh/jiayisunx/61/orig        -> origin/gh/jiayisunx/61/orig
2025-09-07T07:36:03.9838623Z  * [new branch]              gh/jiayisunx/64/base        -> origin/gh/jiayisunx/64/base
2025-09-07T07:36:03.9839849Z  * [new branch]              gh/jiayisunx/64/head        -> origin/gh/jiayisunx/64/head
2025-09-07T07:36:03.9841006Z  * [new branch]              gh/jiayisunx/64/orig        -> origin/gh/jiayisunx/64/orig
2025-09-07T07:36:03.9842711Z  * [new branch]              gh/jiayisunx/65/base        -> origin/gh/jiayisunx/65/base
2025-09-07T07:36:03.9843964Z  * [new branch]              gh/jiayisunx/65/head        -> origin/gh/jiayisunx/65/head
2025-09-07T07:36:03.9845168Z  * [new branch]              gh/jiayisunx/65/orig        -> origin/gh/jiayisunx/65/orig
2025-09-07T07:36:03.9846799Z  * [new branch]              gh/jiayisunx/66/base        -> origin/gh/jiayisunx/66/base
2025-09-07T07:36:03.9847991Z  * [new branch]              gh/jiayisunx/66/head        -> origin/gh/jiayisunx/66/head
2025-09-07T07:36:03.9849137Z  * [new branch]              gh/jiayisunx/66/orig        -> origin/gh/jiayisunx/66/orig
2025-09-07T07:36:03.9850810Z  * [new branch]              gh/jiayisunx/67/base        -> origin/gh/jiayisunx/67/base
2025-09-07T07:36:03.9851961Z  * [new branch]              gh/jiayisunx/67/head        -> origin/gh/jiayisunx/67/head
2025-09-07T07:36:03.9853140Z  * [new branch]              gh/jiayisunx/67/orig        -> origin/gh/jiayisunx/67/orig
2025-09-07T07:36:03.9854942Z  * [new branch]              gh/jiayisunx/68/base        -> origin/gh/jiayisunx/68/base
2025-09-07T07:36:03.9856118Z  * [new branch]              gh/jiayisunx/68/head        -> origin/gh/jiayisunx/68/head
2025-09-07T07:36:03.9857288Z  * [new branch]              gh/jiayisunx/68/orig        -> origin/gh/jiayisunx/68/orig
2025-09-07T07:36:03.9859156Z  * [new branch]              gh/jiayisunx/69/base        -> origin/gh/jiayisunx/69/base
2025-09-07T07:36:03.9860459Z  * [new branch]              gh/jiayisunx/69/head        -> origin/gh/jiayisunx/69/head
2025-09-07T07:36:03.9861675Z  * [new branch]              gh/jiayisunx/69/orig        -> origin/gh/jiayisunx/69/orig
2025-09-07T07:36:03.9863413Z  * [new branch]              gh/jiayisunx/70/base        -> origin/gh/jiayisunx/70/base
2025-09-07T07:36:03.9864587Z  * [new branch]              gh/jiayisunx/70/head        -> origin/gh/jiayisunx/70/head
2025-09-07T07:36:03.9865788Z  * [new branch]              gh/jiayisunx/70/orig        -> origin/gh/jiayisunx/70/orig
2025-09-07T07:36:03.9867486Z  * [new branch]              gh/jiayisunx/71/base        -> origin/gh/jiayisunx/71/base
2025-09-07T07:36:03.9868672Z  * [new branch]              gh/jiayisunx/71/head        -> origin/gh/jiayisunx/71/head
2025-09-07T07:36:03.9869853Z  * [new branch]              gh/jiayisunx/71/orig        -> origin/gh/jiayisunx/71/orig
2025-09-07T07:36:03.9871652Z  * [new branch]              gh/jiayisunx/72/base        -> origin/gh/jiayisunx/72/base
2025-09-07T07:36:03.9872842Z  * [new branch]              gh/jiayisunx/72/head        -> origin/gh/jiayisunx/72/head
2025-09-07T07:36:03.9874061Z  * [new branch]              gh/jiayisunx/72/orig        -> origin/gh/jiayisunx/72/orig
2025-09-07T07:36:03.9875780Z  * [new branch]              gh/jiayisunx/73/base        -> origin/gh/jiayisunx/73/base
2025-09-07T07:36:03.9877045Z  * [new branch]              gh/jiayisunx/73/head        -> origin/gh/jiayisunx/73/head
2025-09-07T07:36:03.9878245Z  * [new branch]              gh/jiayisunx/73/orig        -> origin/gh/jiayisunx/73/orig
2025-09-07T07:36:03.9879894Z  * [new branch]              gh/jiayisunx/74/base        -> origin/gh/jiayisunx/74/base
2025-09-07T07:36:03.9881106Z  * [new branch]              gh/jiayisunx/74/head        -> origin/gh/jiayisunx/74/head
2025-09-07T07:36:03.9882276Z  * [new branch]              gh/jiayisunx/74/orig        -> origin/gh/jiayisunx/74/orig
2025-09-07T07:36:03.9884077Z  * [new branch]              gh/jiayisunx/75/base        -> origin/gh/jiayisunx/75/base
2025-09-07T07:36:03.9885194Z  * [new branch]              gh/jiayisunx/75/head        -> origin/gh/jiayisunx/75/head
2025-09-07T07:36:03.9886248Z  * [new branch]              gh/jiayisunx/75/orig        -> origin/gh/jiayisunx/75/orig
2025-09-07T07:36:03.9887931Z  * [new branch]              gh/jiayisunx/76/base        -> origin/gh/jiayisunx/76/base
2025-09-07T07:36:03.9889049Z  * [new branch]              gh/jiayisunx/76/head        -> origin/gh/jiayisunx/76/head
2025-09-07T07:36:03.9890236Z  * [new branch]              gh/jiayisunx/76/orig        -> origin/gh/jiayisunx/76/orig
2025-09-07T07:36:03.9892255Z  * [new branch]              gh/jjwu@meta.com/1/base     -> origin/gh/jjwu@meta.com/1/base
2025-09-07T07:36:03.9893437Z  * [new branch]              gh/jjwu@meta.com/1/head     -> origin/gh/jjwu@meta.com/1/head
2025-09-07T07:36:03.9895525Z  * [new branch]              gh/justinchuby/111/base     -> origin/gh/justinchuby/111/base
2025-09-07T07:36:03.9896940Z  * [new branch]              gh/justinchuby/111/head     -> origin/gh/justinchuby/111/head
2025-09-07T07:36:03.9898264Z  * [new branch]              gh/justinchuby/111/orig     -> origin/gh/justinchuby/111/orig
2025-09-07T07:36:03.9900004Z  * [new branch]              gh/justinchuby/112/base     -> origin/gh/justinchuby/112/base
2025-09-07T07:36:03.9901188Z  * [new branch]              gh/justinchuby/112/head     -> origin/gh/justinchuby/112/head
2025-09-07T07:36:03.9902543Z  * [new branch]              gh/justinchuby/112/orig     -> origin/gh/justinchuby/112/orig
2025-09-07T07:36:03.9904436Z  * [new branch]              gh/justinchuby/113/base     -> origin/gh/justinchuby/113/base
2025-09-07T07:36:03.9905655Z  * [new branch]              gh/justinchuby/113/head     -> origin/gh/justinchuby/113/head
2025-09-07T07:36:03.9906890Z  * [new branch]              gh/justinchuby/113/orig     -> origin/gh/justinchuby/113/orig
2025-09-07T07:36:03.9908461Z  * [new branch]              gh/justinchuby/114/base     -> origin/gh/justinchuby/114/base
2025-09-07T07:36:03.9909708Z  * [new branch]              gh/justinchuby/114/head     -> origin/gh/justinchuby/114/head
2025-09-07T07:36:03.9910873Z  * [new branch]              gh/justinchuby/114/orig     -> origin/gh/justinchuby/114/orig
2025-09-07T07:36:03.9912541Z  * [new branch]              gh/justinchuby/115/base     -> origin/gh/justinchuby/115/base
2025-09-07T07:36:03.9913720Z  * [new branch]              gh/justinchuby/115/head     -> origin/gh/justinchuby/115/head
2025-09-07T07:36:03.9914824Z  * [new branch]              gh/justinchuby/115/orig     -> origin/gh/justinchuby/115/orig
2025-09-07T07:36:03.9916921Z  * [new branch]              gh/karthickai/1/base        -> origin/gh/karthickai/1/base
2025-09-07T07:36:03.9918337Z  * [new branch]              gh/karthickai/1/head        -> origin/gh/karthickai/1/head
2025-09-07T07:36:03.9919496Z  * [new branch]              gh/karthickai/1/orig        -> origin/gh/karthickai/1/orig
2025-09-07T07:36:03.9921183Z  * [new branch]              gh/karthickai/2/base        -> origin/gh/karthickai/2/base
2025-09-07T07:36:03.9922362Z  * [new branch]              gh/karthickai/2/head        -> origin/gh/karthickai/2/head
2025-09-07T07:36:03.9923538Z  * [new branch]              gh/karthickai/2/orig        -> origin/gh/karthickai/2/orig
2025-09-07T07:36:03.9925613Z  * [new branch]              gh/kurtamohler/32/base      -> origin/gh/kurtamohler/32/base
2025-09-07T07:36:03.9926779Z  * [new branch]              gh/kurtamohler/32/head      -> origin/gh/kurtamohler/32/head
2025-09-07T07:36:03.9928033Z  * [new branch]              gh/kurtamohler/32/orig      -> origin/gh/kurtamohler/32/orig
2025-09-07T07:36:03.9929691Z  * [new branch]              gh/kurtamohler/33/base      -> origin/gh/kurtamohler/33/base
2025-09-07T07:36:03.9930916Z  * [new branch]              gh/kurtamohler/33/head      -> origin/gh/kurtamohler/33/head
2025-09-07T07:36:03.9932211Z  * [new branch]              gh/kurtamohler/33/orig      -> origin/gh/kurtamohler/33/orig
2025-09-07T07:36:03.9934031Z  * [new branch]              gh/kurtamohler/34/base      -> origin/gh/kurtamohler/34/base
2025-09-07T07:36:03.9935278Z  * [new branch]              gh/kurtamohler/34/head      -> origin/gh/kurtamohler/34/head
2025-09-07T07:36:03.9936383Z  * [new branch]              gh/kurtamohler/34/orig      -> origin/gh/kurtamohler/34/orig
2025-09-07T07:36:03.9938143Z  * [new branch]              gh/kurtamohler/41/base      -> origin/gh/kurtamohler/41/base
2025-09-07T07:36:03.9939338Z  * [new branch]              gh/kurtamohler/41/head      -> origin/gh/kurtamohler/41/head
2025-09-07T07:36:03.9940509Z  * [new branch]              gh/kurtamohler/41/orig      -> origin/gh/kurtamohler/41/orig
2025-09-07T07:36:03.9942206Z  * [new branch]              gh/kurtamohler/46/base      -> origin/gh/kurtamohler/46/base
2025-09-07T07:36:03.9943411Z  * [new branch]              gh/kurtamohler/46/head      -> origin/gh/kurtamohler/46/head
2025-09-07T07:36:03.9944658Z  * [new branch]              gh/kurtamohler/46/orig      -> origin/gh/kurtamohler/46/orig
2025-09-07T07:36:03.9946363Z  * [new branch]              gh/kurtamohler/47/base      -> origin/gh/kurtamohler/47/base
2025-09-07T07:36:03.9947669Z  * [new branch]              gh/kurtamohler/47/head      -> origin/gh/kurtamohler/47/head
2025-09-07T07:36:03.9948857Z  * [new branch]              gh/kurtamohler/47/orig      -> origin/gh/kurtamohler/47/orig
2025-09-07T07:36:03.9950604Z  * [new branch]              gh/kurtamohler/48/base      -> origin/gh/kurtamohler/48/base
2025-09-07T07:36:03.9951801Z  * [new branch]              gh/kurtamohler/48/head      -> origin/gh/kurtamohler/48/head
2025-09-07T07:36:03.9952975Z  * [new branch]              gh/kurtamohler/48/orig      -> origin/gh/kurtamohler/48/orig
2025-09-07T07:36:03.9954643Z  * [new branch]              gh/kurtamohler/49/base      -> origin/gh/kurtamohler/49/base
2025-09-07T07:36:03.9955841Z  * [new branch]              gh/kurtamohler/49/head      -> origin/gh/kurtamohler/49/head
2025-09-07T07:36:03.9957030Z  * [new branch]              gh/kurtamohler/49/orig      -> origin/gh/kurtamohler/49/orig
2025-09-07T07:36:03.9958716Z  * [new branch]              gh/kurtamohler/50/base      -> origin/gh/kurtamohler/50/base
2025-09-07T07:36:03.9959929Z  * [new branch]              gh/kurtamohler/50/head      -> origin/gh/kurtamohler/50/head
2025-09-07T07:36:03.9961091Z  * [new branch]              gh/kurtamohler/50/orig      -> origin/gh/kurtamohler/50/orig
2025-09-07T07:36:03.9963339Z  * [new branch]              gh/kwen2501/130/base        -> origin/gh/kwen2501/130/base
2025-09-07T07:36:03.9964648Z  * [new branch]              gh/kwen2501/130/head        -> origin/gh/kwen2501/130/head
2025-09-07T07:36:03.9965870Z  * [new branch]              gh/kwen2501/130/orig        -> origin/gh/kwen2501/130/orig
2025-09-07T07:36:03.9967743Z  * [new branch]              gh/kwen2501/15/base         -> origin/gh/kwen2501/15/base
2025-09-07T07:36:03.9968916Z  * [new branch]              gh/kwen2501/15/head         -> origin/gh/kwen2501/15/head
2025-09-07T07:36:03.9970611Z  * [new branch]              gh/kwen2501/156/base        -> origin/gh/kwen2501/156/base
2025-09-07T07:36:03.9971777Z  * [new branch]              gh/kwen2501/156/head        -> origin/gh/kwen2501/156/head
2025-09-07T07:36:03.9972949Z  * [new branch]              gh/kwen2501/156/orig        -> origin/gh/kwen2501/156/orig
2025-09-07T07:36:03.9974678Z  * [new branch]              gh/kwen2501/170/base        -> origin/gh/kwen2501/170/base
2025-09-07T07:36:03.9975851Z  * [new branch]              gh/kwen2501/170/head        -> origin/gh/kwen2501/170/head
2025-09-07T07:36:03.9977597Z  * [new branch]              gh/kwen2501/186/base        -> origin/gh/kwen2501/186/base
2025-09-07T07:36:03.9978894Z  * [new branch]              gh/kwen2501/186/head        -> origin/gh/kwen2501/186/head
2025-09-07T07:36:03.9980078Z  * [new branch]              gh/kwen2501/186/orig        -> origin/gh/kwen2501/186/orig
2025-09-07T07:36:03.9981673Z  * [new branch]              gh/kwen2501/187/base        -> origin/gh/kwen2501/187/base
2025-09-07T07:36:03.9983616Z  * [new branch]              gh/kwen2501/187/head        -> origin/gh/kwen2501/187/head
2025-09-07T07:36:03.9984868Z  * [new branch]              gh/kwen2501/187/orig        -> origin/gh/kwen2501/187/orig
2025-09-07T07:36:03.9986561Z  * [new branch]              gh/kwen2501/188/base        -> origin/gh/kwen2501/188/base
2025-09-07T07:36:03.9987728Z  * [new branch]              gh/kwen2501/188/head        -> origin/gh/kwen2501/188/head
2025-09-07T07:36:03.9988918Z  * [new branch]              gh/kwen2501/188/orig        -> origin/gh/kwen2501/188/orig
2025-09-07T07:36:03.9990683Z  * [new branch]              gh/kwen2501/194/base        -> origin/gh/kwen2501/194/base
2025-09-07T07:36:03.9991863Z  * [new branch]              gh/kwen2501/194/head        -> origin/gh/kwen2501/194/head
2025-09-07T07:36:03.9993088Z  * [new branch]              gh/kwen2501/194/orig        -> origin/gh/kwen2501/194/orig
2025-09-07T07:36:03.9994770Z  * [new branch]              gh/kwen2501/199/base        -> origin/gh/kwen2501/199/base
2025-09-07T07:36:03.9995964Z  * [new branch]              gh/kwen2501/199/head        -> origin/gh/kwen2501/199/head
2025-09-07T07:36:03.9997125Z  * [new branch]              gh/kwen2501/199/orig        -> origin/gh/kwen2501/199/orig
2025-09-07T07:36:03.9998732Z  * [new branch]              gh/kwen2501/200/base        -> origin/gh/kwen2501/200/base
2025-09-07T07:36:04.0000042Z  * [new branch]              gh/kwen2501/200/head        -> origin/gh/kwen2501/200/head
2025-09-07T07:36:04.0001210Z  * [new branch]              gh/kwen2501/200/orig        -> origin/gh/kwen2501/200/orig
2025-09-07T07:36:04.0003007Z  * [new branch]              gh/kwen2501/201/base        -> origin/gh/kwen2501/201/base
2025-09-07T07:36:04.0004396Z  * [new branch]              gh/kwen2501/201/head        -> origin/gh/kwen2501/201/head
2025-09-07T07:36:04.0005599Z  * [new branch]              gh/kwen2501/201/orig        -> origin/gh/kwen2501/201/orig
2025-09-07T07:36:04.0007299Z  * [new branch]              gh/kwen2501/203/base        -> origin/gh/kwen2501/203/base
2025-09-07T07:36:04.0008479Z  * [new branch]              gh/kwen2501/203/head        -> origin/gh/kwen2501/203/head
2025-09-07T07:36:04.0009645Z  * [new branch]              gh/kwen2501/203/orig        -> origin/gh/kwen2501/203/orig
2025-09-07T07:36:04.0011366Z  * [new branch]              gh/kwen2501/204/base        -> origin/gh/kwen2501/204/base
2025-09-07T07:36:04.0012512Z  * [new branch]              gh/kwen2501/204/head        -> origin/gh/kwen2501/204/head
2025-09-07T07:36:04.0013698Z  * [new branch]              gh/kwen2501/204/orig        -> origin/gh/kwen2501/204/orig
2025-09-07T07:36:04.0015355Z  * [new branch]              gh/kwen2501/205/base        -> origin/gh/kwen2501/205/base
2025-09-07T07:36:04.0016585Z  * [new branch]              gh/kwen2501/205/head        -> origin/gh/kwen2501/205/head
2025-09-07T07:36:04.0017774Z  * [new branch]              gh/kwen2501/205/orig        -> origin/gh/kwen2501/205/orig
2025-09-07T07:36:04.0019524Z  * [new branch]              gh/kwen2501/206/base        -> origin/gh/kwen2501/206/base
2025-09-07T07:36:04.0020721Z  * [new branch]              gh/kwen2501/206/head        -> origin/gh/kwen2501/206/head
2025-09-07T07:36:04.0021898Z  * [new branch]              gh/kwen2501/206/orig        -> origin/gh/kwen2501/206/orig
2025-09-07T07:36:04.0023590Z  * [new branch]              gh/kwen2501/207/base        -> origin/gh/kwen2501/207/base
2025-09-07T07:36:04.0024755Z  * [new branch]              gh/kwen2501/207/head        -> origin/gh/kwen2501/207/head
2025-09-07T07:36:04.0026051Z  * [new branch]              gh/kwen2501/207/orig        -> origin/gh/kwen2501/207/orig
2025-09-07T07:36:04.0027739Z  * [new branch]              gh/kwen2501/208/base        -> origin/gh/kwen2501/208/base
2025-09-07T07:36:04.0028923Z  * [new branch]              gh/kwen2501/208/head        -> origin/gh/kwen2501/208/head
2025-09-07T07:36:04.0030098Z  * [new branch]              gh/kwen2501/208/orig        -> origin/gh/kwen2501/208/orig
2025-09-07T07:36:04.0032342Z  * [new branch]              gh/kwen2501/209/base        -> origin/gh/kwen2501/209/base
2025-09-07T07:36:04.0033494Z  * [new branch]              gh/kwen2501/209/head        -> origin/gh/kwen2501/209/head
2025-09-07T07:36:04.0034672Z  * [new branch]              gh/kwen2501/209/orig        -> origin/gh/kwen2501/209/orig
2025-09-07T07:36:04.0036430Z  * [new branch]              gh/kwen2501/210/base        -> origin/gh/kwen2501/210/base
2025-09-07T07:36:04.0037743Z  * [new branch]              gh/kwen2501/210/head        -> origin/gh/kwen2501/210/head
2025-09-07T07:36:04.0038923Z  * [new branch]              gh/kwen2501/210/orig        -> origin/gh/kwen2501/210/orig
2025-09-07T07:36:04.0040650Z  * [new branch]              gh/kwen2501/211/base        -> origin/gh/kwen2501/211/base
2025-09-07T07:36:04.0041848Z  * [new branch]              gh/kwen2501/211/head        -> origin/gh/kwen2501/211/head
2025-09-07T07:36:04.0043631Z  * [new branch]              gh/kwen2501/212/base        -> origin/gh/kwen2501/212/base
2025-09-07T07:36:04.0044827Z  * [new branch]              gh/kwen2501/212/head        -> origin/gh/kwen2501/212/head
2025-09-07T07:36:04.0046008Z  * [new branch]              gh/kwen2501/212/orig        -> origin/gh/kwen2501/212/orig
2025-09-07T07:36:04.0047738Z  * [new branch]              gh/kwen2501/213/base        -> origin/gh/kwen2501/213/base
2025-09-07T07:36:04.0048933Z  * [new branch]              gh/kwen2501/213/head        -> origin/gh/kwen2501/213/head
2025-09-07T07:36:04.0050156Z  * [new branch]              gh/kwen2501/213/orig        -> origin/gh/kwen2501/213/orig
2025-09-07T07:36:04.0051906Z  * [new branch]              gh/kwen2501/214/base        -> origin/gh/kwen2501/214/base
2025-09-07T07:36:04.0053098Z  * [new branch]              gh/kwen2501/214/head        -> origin/gh/kwen2501/214/head
2025-09-07T07:36:04.0054274Z  * [new branch]              gh/kwen2501/214/orig        -> origin/gh/kwen2501/214/orig
2025-09-07T07:36:04.0056050Z  * [new branch]              gh/kwen2501/215/base        -> origin/gh/kwen2501/215/base
2025-09-07T07:36:04.0057259Z  * [new branch]              gh/kwen2501/215/head        -> origin/gh/kwen2501/215/head
2025-09-07T07:36:04.0058539Z  * [new branch]              gh/kwen2501/215/orig        -> origin/gh/kwen2501/215/orig
2025-09-07T07:36:04.0060290Z  * [new branch]              gh/kwen2501/216/base        -> origin/gh/kwen2501/216/base
2025-09-07T07:36:04.0061501Z  * [new branch]              gh/kwen2501/216/head        -> origin/gh/kwen2501/216/head
2025-09-07T07:36:04.0062722Z  * [new branch]              gh/kwen2501/216/orig        -> origin/gh/kwen2501/216/orig
2025-09-07T07:36:04.0064417Z  * [new branch]              gh/kwen2501/217/base        -> origin/gh/kwen2501/217/base
2025-09-07T07:36:04.0065605Z  * [new branch]              gh/kwen2501/217/head        -> origin/gh/kwen2501/217/head
2025-09-07T07:36:04.0066904Z  * [new branch]              gh/kwen2501/217/orig        -> origin/gh/kwen2501/217/orig
2025-09-07T07:36:04.0068616Z  * [new branch]              gh/kwen2501/218/base        -> origin/gh/kwen2501/218/base
2025-09-07T07:36:04.0069848Z  * [new branch]              gh/kwen2501/218/head        -> origin/gh/kwen2501/218/head
2025-09-07T07:36:04.0071012Z  * [new branch]              gh/kwen2501/218/orig        -> origin/gh/kwen2501/218/orig
2025-09-07T07:36:04.0072844Z  * [new branch]              gh/kwen2501/219/base        -> origin/gh/kwen2501/219/base
2025-09-07T07:36:04.0074002Z  * [new branch]              gh/kwen2501/219/head        -> origin/gh/kwen2501/219/head
2025-09-07T07:36:04.0075157Z  * [new branch]              gh/kwen2501/219/orig        -> origin/gh/kwen2501/219/orig
2025-09-07T07:36:04.0076880Z  * [new branch]              gh/kwen2501/220/base        -> origin/gh/kwen2501/220/base
2025-09-07T07:36:04.0078069Z  * [new branch]              gh/kwen2501/220/head        -> origin/gh/kwen2501/220/head
2025-09-07T07:36:04.0079212Z  * [new branch]              gh/kwen2501/220/orig        -> origin/gh/kwen2501/220/orig
2025-09-07T07:36:04.0080998Z  * [new branch]              gh/kwen2501/221/base        -> origin/gh/kwen2501/221/base
2025-09-07T07:36:04.0082630Z  * [new branch]              gh/kwen2501/221/head        -> origin/gh/kwen2501/221/head
2025-09-07T07:36:04.0083909Z  * [new branch]              gh/kwen2501/221/orig        -> origin/gh/kwen2501/221/orig
2025-09-07T07:36:04.0085674Z  * [new branch]              gh/kwen2501/222/base        -> origin/gh/kwen2501/222/base
2025-09-07T07:36:04.0086875Z  * [new branch]              gh/kwen2501/222/head        -> origin/gh/kwen2501/222/head
2025-09-07T07:36:04.0088034Z  * [new branch]              gh/kwen2501/222/orig        -> origin/gh/kwen2501/222/orig
2025-09-07T07:36:04.0089752Z  * [new branch]              gh/kwen2501/223/base        -> origin/gh/kwen2501/223/base
2025-09-07T07:36:04.0090919Z  * [new branch]              gh/kwen2501/223/head        -> origin/gh/kwen2501/223/head
2025-09-07T07:36:04.0092061Z  * [new branch]              gh/kwen2501/223/orig        -> origin/gh/kwen2501/223/orig
2025-09-07T07:36:04.0093756Z  * [new branch]              gh/kwen2501/224/base        -> origin/gh/kwen2501/224/base
2025-09-07T07:36:04.0094909Z  * [new branch]              gh/kwen2501/224/head        -> origin/gh/kwen2501/224/head
2025-09-07T07:36:04.0096079Z  * [new branch]              gh/kwen2501/224/orig        -> origin/gh/kwen2501/224/orig
2025-09-07T07:36:04.0097915Z  * [new branch]              gh/kwen2501/225/base        -> origin/gh/kwen2501/225/base
2025-09-07T07:36:04.0099132Z  * [new branch]              gh/kwen2501/225/head        -> origin/gh/kwen2501/225/head
2025-09-07T07:36:04.0100383Z  * [new branch]              gh/kwen2501/225/orig        -> origin/gh/kwen2501/225/orig
2025-09-07T07:36:04.0102047Z  * [new branch]              gh/kwen2501/226/base        -> origin/gh/kwen2501/226/base
2025-09-07T07:36:04.0103384Z  * [new branch]              gh/kwen2501/226/head        -> origin/gh/kwen2501/226/head
2025-09-07T07:36:04.0107910Z  * [new branch]              gh/kwen2501/226/orig        -> origin/gh/kwen2501/226/orig
2025-09-07T07:36:04.0109560Z  * [new branch]              gh/kwen2501/227/base        -> origin/gh/kwen2501/227/base
2025-09-07T07:36:04.0110940Z  * [new branch]              gh/kwen2501/227/head        -> origin/gh/kwen2501/227/head
2025-09-07T07:36:04.0112151Z  * [new branch]              gh/kwen2501/227/orig        -> origin/gh/kwen2501/227/orig
2025-09-07T07:36:04.0113840Z  * [new branch]              gh/kwen2501/228/base        -> origin/gh/kwen2501/228/base
2025-09-07T07:36:04.0115001Z  * [new branch]              gh/kwen2501/228/head        -> origin/gh/kwen2501/228/head
2025-09-07T07:36:04.0116162Z  * [new branch]              gh/kwen2501/228/orig        -> origin/gh/kwen2501/228/orig
2025-09-07T07:36:04.0117893Z  * [new branch]              gh/kwen2501/229/base        -> origin/gh/kwen2501/229/base
2025-09-07T07:36:04.0119119Z  * [new branch]              gh/kwen2501/229/head        -> origin/gh/kwen2501/229/head
2025-09-07T07:36:04.0120380Z  * [new branch]              gh/kwen2501/229/orig        -> origin/gh/kwen2501/229/orig
2025-09-07T07:36:04.0122125Z  * [new branch]              gh/kwen2501/230/base        -> origin/gh/kwen2501/230/base
2025-09-07T07:36:04.0123292Z  * [new branch]              gh/kwen2501/230/head        -> origin/gh/kwen2501/230/head
2025-09-07T07:36:04.0124484Z  * [new branch]              gh/kwen2501/230/orig        -> origin/gh/kwen2501/230/orig
2025-09-07T07:36:04.0126198Z  * [new branch]              gh/kwen2501/231/base        -> origin/gh/kwen2501/231/base
2025-09-07T07:36:04.0127375Z  * [new branch]              gh/kwen2501/231/head        -> origin/gh/kwen2501/231/head
2025-09-07T07:36:04.0128591Z  * [new branch]              gh/kwen2501/231/orig        -> origin/gh/kwen2501/231/orig
2025-09-07T07:36:04.0130298Z  * [new branch]              gh/kwen2501/232/base        -> origin/gh/kwen2501/232/base
2025-09-07T07:36:04.0131480Z  * [new branch]              gh/kwen2501/232/head        -> origin/gh/kwen2501/232/head
2025-09-07T07:36:04.0132615Z  * [new branch]              gh/kwen2501/232/orig        -> origin/gh/kwen2501/232/orig
2025-09-07T07:36:04.0134975Z  * [new branch]              gh/laithsakka/156/base      -> origin/gh/laithsakka/156/base
2025-09-07T07:36:04.0136498Z  * [new branch]              gh/laithsakka/156/head      -> origin/gh/laithsakka/156/head
2025-09-07T07:36:04.0137776Z  * [new branch]              gh/laithsakka/156/orig      -> origin/gh/laithsakka/156/orig
2025-09-07T07:36:04.0139588Z  * [new branch]              gh/laithsakka/160/base      -> origin/gh/laithsakka/160/base
2025-09-07T07:36:04.0140775Z  * [new branch]              gh/laithsakka/160/head      -> origin/gh/laithsakka/160/head
2025-09-07T07:36:04.0141945Z  * [new branch]              gh/laithsakka/160/orig      -> origin/gh/laithsakka/160/orig
2025-09-07T07:36:04.0143639Z  * [new branch]              gh/laithsakka/178/base      -> origin/gh/laithsakka/178/base
2025-09-07T07:36:04.0144850Z  * [new branch]              gh/laithsakka/178/head      -> origin/gh/laithsakka/178/head
2025-09-07T07:36:04.0146040Z  * [new branch]              gh/laithsakka/178/orig      -> origin/gh/laithsakka/178/orig
2025-09-07T07:36:04.0147861Z  * [new branch]              gh/laithsakka/191/base      -> origin/gh/laithsakka/191/base
2025-09-07T07:36:04.0149088Z  * [new branch]              gh/laithsakka/191/head      -> origin/gh/laithsakka/191/head
2025-09-07T07:36:04.0150258Z  * [new branch]              gh/laithsakka/191/orig      -> origin/gh/laithsakka/191/orig
2025-09-07T07:36:04.0152054Z  * [new branch]              gh/laithsakka/237/base      -> origin/gh/laithsakka/237/base
2025-09-07T07:36:04.0153292Z  * [new branch]              gh/laithsakka/237/head      -> origin/gh/laithsakka/237/head
2025-09-07T07:36:04.0154466Z  * [new branch]              gh/laithsakka/237/orig      -> origin/gh/laithsakka/237/orig
2025-09-07T07:36:04.0156171Z  * [new branch]              gh/laithsakka/249/base      -> origin/gh/laithsakka/249/base
2025-09-07T07:36:04.0157788Z  * [new branch]              gh/laithsakka/249/head      -> origin/gh/laithsakka/249/head
2025-09-07T07:36:04.0158993Z  * [new branch]              gh/laithsakka/249/orig      -> origin/gh/laithsakka/249/orig
2025-09-07T07:36:04.0160717Z  * [new branch]              gh/laithsakka/251/base      -> origin/gh/laithsakka/251/base
2025-09-07T07:36:04.0161910Z  * [new branch]              gh/laithsakka/251/head      -> origin/gh/laithsakka/251/head
2025-09-07T07:36:04.0163100Z  * [new branch]              gh/laithsakka/251/orig      -> origin/gh/laithsakka/251/orig
2025-09-07T07:36:04.0164858Z  * [new branch]              gh/laithsakka/254/base      -> origin/gh/laithsakka/254/base
2025-09-07T07:36:04.0165976Z  * [new branch]              gh/laithsakka/254/head      -> origin/gh/laithsakka/254/head
2025-09-07T07:36:04.0167174Z  * [new branch]              gh/laithsakka/254/orig      -> origin/gh/laithsakka/254/orig
2025-09-07T07:36:04.0169015Z  * [new branch]              gh/laithsakka/255/base      -> origin/gh/laithsakka/255/base
2025-09-07T07:36:04.0170178Z  * [new branch]              gh/laithsakka/255/head      -> origin/gh/laithsakka/255/head
2025-09-07T07:36:04.0171316Z  * [new branch]              gh/laithsakka/255/orig      -> origin/gh/laithsakka/255/orig
2025-09-07T07:36:04.0173005Z  * [new branch]              gh/laithsakka/256/base      -> origin/gh/laithsakka/256/base
2025-09-07T07:36:04.0174244Z  * [new branch]              gh/laithsakka/256/head      -> origin/gh/laithsakka/256/head
2025-09-07T07:36:04.0175317Z  * [new branch]              gh/laithsakka/256/orig      -> origin/gh/laithsakka/256/orig
2025-09-07T07:36:04.0176998Z  * [new branch]              gh/laithsakka/257/base      -> origin/gh/laithsakka/257/base
2025-09-07T07:36:04.0178251Z  * [new branch]              gh/laithsakka/257/head      -> origin/gh/laithsakka/257/head
2025-09-07T07:36:04.0179454Z  * [new branch]              gh/laithsakka/257/orig      -> origin/gh/laithsakka/257/orig
2025-09-07T07:36:04.0181140Z  * [new branch]              gh/laithsakka/258/base      -> origin/gh/laithsakka/258/base
2025-09-07T07:36:04.0182555Z  * [new branch]              gh/laithsakka/258/head      -> origin/gh/laithsakka/258/head
2025-09-07T07:36:04.0183654Z  * [new branch]              gh/laithsakka/258/orig      -> origin/gh/laithsakka/258/orig
2025-09-07T07:36:04.0185361Z  * [new branch]              gh/laithsakka/259/base      -> origin/gh/laithsakka/259/base
2025-09-07T07:36:04.0186585Z  * [new branch]              gh/laithsakka/259/head      -> origin/gh/laithsakka/259/head
2025-09-07T07:36:04.0187776Z  * [new branch]              gh/laithsakka/259/orig      -> origin/gh/laithsakka/259/orig
2025-09-07T07:36:04.0189384Z  * [new branch]              gh/laithsakka/260/base      -> origin/gh/laithsakka/260/base
2025-09-07T07:36:04.0190564Z  * [new branch]              gh/laithsakka/260/head      -> origin/gh/laithsakka/260/head
2025-09-07T07:36:04.0191907Z  * [new branch]              gh/laithsakka/260/orig      -> origin/gh/laithsakka/260/orig
2025-09-07T07:36:04.0193575Z  * [new branch]              gh/laithsakka/261/base      -> origin/gh/laithsakka/261/base
2025-09-07T07:36:04.0194762Z  * [new branch]              gh/laithsakka/261/head      -> origin/gh/laithsakka/261/head
2025-09-07T07:36:04.0195899Z  * [new branch]              gh/laithsakka/261/orig      -> origin/gh/laithsakka/261/orig
2025-09-07T07:36:04.0197873Z  * [new branch]              gh/laithsakka/262/base      -> origin/gh/laithsakka/262/base
2025-09-07T07:36:04.0199352Z  * [new branch]              gh/laithsakka/262/head      -> origin/gh/laithsakka/262/head
2025-09-07T07:36:04.0200603Z  * [new branch]              gh/laithsakka/262/orig      -> origin/gh/laithsakka/262/orig
2025-09-07T07:36:04.0202688Z  * [new branch]              gh/laithsakka/263/base      -> origin/gh/laithsakka/263/base
2025-09-07T07:36:04.0204187Z  * [new branch]              gh/laithsakka/263/head      -> origin/gh/laithsakka/263/head
2025-09-07T07:36:04.0205398Z  * [new branch]              gh/laithsakka/263/orig      -> origin/gh/laithsakka/263/orig
2025-09-07T07:36:04.0207081Z  * [new branch]              gh/laithsakka/264/base      -> origin/gh/laithsakka/264/base
2025-09-07T07:36:04.0208254Z  * [new branch]              gh/laithsakka/264/head      -> origin/gh/laithsakka/264/head
2025-09-07T07:36:04.0209410Z  * [new branch]              gh/laithsakka/264/orig      -> origin/gh/laithsakka/264/orig
2025-09-07T07:36:04.0211209Z  * [new branch]              gh/laithsakka/265/base      -> origin/gh/laithsakka/265/base
2025-09-07T07:36:04.0212442Z  * [new branch]              gh/laithsakka/265/head      -> origin/gh/laithsakka/265/head
2025-09-07T07:36:04.0213633Z  * [new branch]              gh/laithsakka/265/orig      -> origin/gh/laithsakka/265/orig
2025-09-07T07:36:04.0215369Z  * [new branch]              gh/laithsakka/266/base      -> origin/gh/laithsakka/266/base
2025-09-07T07:36:04.0216576Z  * [new branch]              gh/laithsakka/266/head      -> origin/gh/laithsakka/266/head
2025-09-07T07:36:04.0218016Z  * [new branch]              gh/laithsakka/266/orig      -> origin/gh/laithsakka/266/orig
2025-09-07T07:36:04.0219780Z  * [new branch]              gh/laithsakka/267/base      -> origin/gh/laithsakka/267/base
2025-09-07T07:36:04.0221434Z  * [new branch]              gh/laithsakka/267/head      -> origin/gh/laithsakka/267/head
2025-09-07T07:36:04.0222617Z  * [new branch]              gh/laithsakka/267/orig      -> origin/gh/laithsakka/267/orig
2025-09-07T07:36:04.0224402Z  * [new branch]              gh/laithsakka/268/base      -> origin/gh/laithsakka/268/base
2025-09-07T07:36:04.0225546Z  * [new branch]              gh/laithsakka/268/head      -> origin/gh/laithsakka/268/head
2025-09-07T07:36:04.0226776Z  * [new branch]              gh/laithsakka/268/orig      -> origin/gh/laithsakka/268/orig
2025-09-07T07:36:04.0228600Z  * [new branch]              gh/laithsakka/28/base       -> origin/gh/laithsakka/28/base
2025-09-07T07:36:04.0230205Z  * [new branch]              gh/laithsakka/29/base       -> origin/gh/laithsakka/29/base
2025-09-07T07:36:04.0232028Z  * [new branch]              gh/laithsakka/30/base       -> origin/gh/laithsakka/30/base
2025-09-07T07:36:04.0233157Z  * [new branch]              gh/laithsakka/30/head       -> origin/gh/laithsakka/30/head
2025-09-07T07:36:04.0234786Z  * [new branch]              gh/laithsakka/31/base       -> origin/gh/laithsakka/31/base
2025-09-07T07:36:04.0235917Z  * [new branch]              gh/laithsakka/31/head       -> origin/gh/laithsakka/31/head
2025-09-07T07:36:04.0237572Z  * [new branch]              gh/laithsakka/32/base       -> origin/gh/laithsakka/32/base
2025-09-07T07:36:04.0238700Z  * [new branch]              gh/laithsakka/32/head       -> origin/gh/laithsakka/32/head
2025-09-07T07:36:04.0242174Z  * [new branch]              gh/lucaskabela/1/base       -> origin/gh/lucaskabela/1/base
2025-09-07T07:36:04.0243354Z  * [new branch]              gh/lucaskabela/1/head       -> origin/gh/lucaskabela/1/head
2025-09-07T07:36:04.0245160Z  * [new branch]              gh/lucaskabela/10/base      -> origin/gh/lucaskabela/10/base
2025-09-07T07:36:04.0246370Z  * [new branch]              gh/lucaskabela/10/head      -> origin/gh/lucaskabela/10/head
2025-09-07T07:36:04.0247549Z  * [new branch]              gh/lucaskabela/10/orig      -> origin/gh/lucaskabela/10/orig
2025-09-07T07:36:04.0249114Z  * [new branch]              gh/lucaskabela/11/base      -> origin/gh/lucaskabela/11/base
2025-09-07T07:36:04.0250308Z  * [new branch]              gh/lucaskabela/11/head      -> origin/gh/lucaskabela/11/head
2025-09-07T07:36:04.0251490Z  * [new branch]              gh/lucaskabela/11/orig      -> origin/gh/lucaskabela/11/orig
2025-09-07T07:36:04.0253168Z  * [new branch]              gh/lucaskabela/12/base      -> origin/gh/lucaskabela/12/base
2025-09-07T07:36:04.0254332Z  * [new branch]              gh/lucaskabela/12/head      -> origin/gh/lucaskabela/12/head
2025-09-07T07:36:04.0255642Z  * [new branch]              gh/lucaskabela/12/orig      -> origin/gh/lucaskabela/12/orig
2025-09-07T07:36:04.0257235Z  * [new branch]              gh/lucaskabela/13/base      -> origin/gh/lucaskabela/13/base
2025-09-07T07:36:04.0258533Z  * [new branch]              gh/lucaskabela/13/head      -> origin/gh/lucaskabela/13/head
2025-09-07T07:36:04.0259710Z  * [new branch]              gh/lucaskabela/13/orig      -> origin/gh/lucaskabela/13/orig
2025-09-07T07:36:04.0261347Z  * [new branch]              gh/lucaskabela/14/base      -> origin/gh/lucaskabela/14/base
2025-09-07T07:36:04.0263018Z  * [new branch]              gh/lucaskabela/14/head      -> origin/gh/lucaskabela/14/head
2025-09-07T07:36:04.0264242Z  * [new branch]              gh/lucaskabela/14/orig      -> origin/gh/lucaskabela/14/orig
2025-09-07T07:36:04.0265853Z  * [new branch]              gh/lucaskabela/15/base      -> origin/gh/lucaskabela/15/base
2025-09-07T07:36:04.0267027Z  * [new branch]              gh/lucaskabela/15/head      -> origin/gh/lucaskabela/15/head
2025-09-07T07:36:04.0268252Z  * [new branch]              gh/lucaskabela/15/orig      -> origin/gh/lucaskabela/15/orig
2025-09-07T07:36:04.0269995Z  * [new branch]              gh/lucaskabela/16/base      -> origin/gh/lucaskabela/16/base
2025-09-07T07:36:04.0271183Z  * [new branch]              gh/lucaskabela/16/head      -> origin/gh/lucaskabela/16/head
2025-09-07T07:36:04.0272364Z  * [new branch]              gh/lucaskabela/16/orig      -> origin/gh/lucaskabela/16/orig
2025-09-07T07:36:04.0273988Z  * [new branch]              gh/lucaskabela/17/base      -> origin/gh/lucaskabela/17/base
2025-09-07T07:36:04.0275096Z  * [new branch]              gh/lucaskabela/17/head      -> origin/gh/lucaskabela/17/head
2025-09-07T07:36:04.0276230Z  * [new branch]              gh/lucaskabela/17/orig      -> origin/gh/lucaskabela/17/orig
2025-09-07T07:36:04.0277922Z  * [new branch]              gh/lucaskabela/2/base       -> origin/gh/lucaskabela/2/base
2025-09-07T07:36:04.0279090Z  * [new branch]              gh/lucaskabela/2/head       -> origin/gh/lucaskabela/2/head
2025-09-07T07:36:04.0280281Z  * [new branch]              gh/lucaskabela/2/orig       -> origin/gh/lucaskabela/2/orig
2025-09-07T07:36:04.0282066Z  * [new branch]              gh/lucaskabela/3/base       -> origin/gh/lucaskabela/3/base
2025-09-07T07:36:04.0283196Z  * [new branch]              gh/lucaskabela/3/head       -> origin/gh/lucaskabela/3/head
2025-09-07T07:36:04.0284352Z  * [new branch]              gh/lucaskabela/3/orig       -> origin/gh/lucaskabela/3/orig
2025-09-07T07:36:04.0286038Z  * [new branch]              gh/lucaskabela/4/base       -> origin/gh/lucaskabela/4/base
2025-09-07T07:36:04.0287208Z  * [new branch]              gh/lucaskabela/4/head       -> origin/gh/lucaskabela/4/head
2025-09-07T07:36:04.0288417Z  * [new branch]              gh/lucaskabela/4/orig       -> origin/gh/lucaskabela/4/orig
2025-09-07T07:36:04.0290264Z  * [new branch]              gh/lucaskabela/5/base       -> origin/gh/lucaskabela/5/base
2025-09-07T07:36:04.0291433Z  * [new branch]              gh/lucaskabela/5/head       -> origin/gh/lucaskabela/5/head
2025-09-07T07:36:04.0292612Z  * [new branch]              gh/lucaskabela/5/orig       -> origin/gh/lucaskabela/5/orig
2025-09-07T07:36:04.0294241Z  * [new branch]              gh/lucaskabela/6/base       -> origin/gh/lucaskabela/6/base
2025-09-07T07:36:04.0295462Z  * [new branch]              gh/lucaskabela/6/head       -> origin/gh/lucaskabela/6/head
2025-09-07T07:36:04.0296656Z  * [new branch]              gh/lucaskabela/6/orig       -> origin/gh/lucaskabela/6/orig
2025-09-07T07:36:04.0298525Z  * [new branch]              gh/lucaskabela/7/base       -> origin/gh/lucaskabela/7/base
2025-09-07T07:36:04.0299675Z  * [new branch]              gh/lucaskabela/7/head       -> origin/gh/lucaskabela/7/head
2025-09-07T07:36:04.0300865Z  * [new branch]              gh/lucaskabela/7/orig       -> origin/gh/lucaskabela/7/orig
2025-09-07T07:36:04.0302554Z  * [new branch]              gh/lucaskabela/8/base       -> origin/gh/lucaskabela/8/base
2025-09-07T07:36:04.0304011Z  * [new branch]              gh/lucaskabela/8/head       -> origin/gh/lucaskabela/8/head
2025-09-07T07:36:04.0305219Z  * [new branch]              gh/lucaskabela/8/orig       -> origin/gh/lucaskabela/8/orig
2025-09-07T07:36:04.0306913Z  * [new branch]              gh/lucaskabela/9/base       -> origin/gh/lucaskabela/9/base
2025-09-07T07:36:04.0308273Z  * [new branch]              gh/lucaskabela/9/head       -> origin/gh/lucaskabela/9/head
2025-09-07T07:36:04.0309234Z  * [new branch]              gh/lucaskabela/9/orig       -> origin/gh/lucaskabela/9/orig
2025-09-07T07:36:04.0311404Z  * [new branch]              gh/lw/3/base                -> origin/gh/lw/3/base
2025-09-07T07:36:04.0312600Z  * [new branch]              gh/lw/3/head                -> origin/gh/lw/3/head
2025-09-07T07:36:04.0313988Z  * [new branch]              gh/lw/3/orig                -> origin/gh/lw/3/orig
2025-09-07T07:36:04.0315910Z  * [new branch]              gh/malfet/14/base           -> origin/gh/malfet/14/base
2025-09-07T07:36:04.0318189Z  * [new branch]              gh/malfet/330/base          -> origin/gh/malfet/330/base
2025-09-07T07:36:04.0319413Z  * [new branch]              gh/malfet/330/head          -> origin/gh/malfet/330/head
2025-09-07T07:36:04.0320602Z  * [new branch]              gh/malfet/330/orig          -> origin/gh/malfet/330/orig
2025-09-07T07:36:04.0322273Z  * [new branch]              gh/malfet/396/base          -> origin/gh/malfet/396/base
2025-09-07T07:36:04.0323464Z  * [new branch]              gh/malfet/396/head          -> origin/gh/malfet/396/head
2025-09-07T07:36:04.0324660Z  * [new branch]              gh/malfet/396/orig          -> origin/gh/malfet/396/orig
2025-09-07T07:36:04.0326475Z  * [new branch]              gh/malfet/397/base          -> origin/gh/malfet/397/base
2025-09-07T07:36:04.0327705Z  * [new branch]              gh/malfet/397/head          -> origin/gh/malfet/397/head
2025-09-07T07:36:04.0328865Z  * [new branch]              gh/malfet/397/orig          -> origin/gh/malfet/397/orig
2025-09-07T07:36:04.0330571Z  * [new branch]              gh/malfet/398/base          -> origin/gh/malfet/398/base
2025-09-07T07:36:04.0331761Z  * [new branch]              gh/malfet/398/head          -> origin/gh/malfet/398/head
2025-09-07T07:36:04.0332866Z  * [new branch]              gh/malfet/398/orig          -> origin/gh/malfet/398/orig
2025-09-07T07:36:04.0334605Z  * [new branch]              gh/malfet/399/base          -> origin/gh/malfet/399/base
2025-09-07T07:36:04.0337945Z  * [new branch]              gh/malfet/399/head          -> origin/gh/malfet/399/head
2025-09-07T07:36:04.0338390Z  * [new branch]              gh/malfet/399/orig          -> origin/gh/malfet/399/orig
2025-09-07T07:36:04.0340631Z  * [new branch]              gh/malfet/414/base          -> origin/gh/malfet/414/base
2025-09-07T07:36:04.0340767Z  * [new branch]              gh/malfet/414/head          -> origin/gh/malfet/414/head
2025-09-07T07:36:04.0341612Z  * [new branch]              gh/malfet/414/orig          -> origin/gh/malfet/414/orig
2025-09-07T07:36:04.0343263Z  * [new branch]              gh/malfet/417/base          -> origin/gh/malfet/417/base
2025-09-07T07:36:04.0344440Z  * [new branch]              gh/malfet/417/head          -> origin/gh/malfet/417/head
2025-09-07T07:36:04.0345647Z  * [new branch]              gh/malfet/417/orig          -> origin/gh/malfet/417/orig
2025-09-07T07:36:04.0347243Z  * [new branch]              gh/malfet/418/base          -> origin/gh/malfet/418/base
2025-09-07T07:36:04.0348393Z  * [new branch]              gh/malfet/418/head          -> origin/gh/malfet/418/head
2025-09-07T07:36:04.0349585Z  * [new branch]              gh/malfet/418/orig          -> origin/gh/malfet/418/orig
2025-09-07T07:36:04.0351384Z  * [new branch]              gh/malfet/475/base          -> origin/gh/malfet/475/base
2025-09-07T07:36:04.0353196Z  * [new branch]              gh/malfet/475/head          -> origin/gh/malfet/475/head
2025-09-07T07:36:04.0354402Z  * [new branch]              gh/malfet/475/orig          -> origin/gh/malfet/475/orig
2025-09-07T07:36:04.0356486Z  * [new branch]              gh/malfet/476/base          -> origin/gh/malfet/476/base
2025-09-07T07:36:04.0357686Z  * [new branch]              gh/malfet/476/head          -> origin/gh/malfet/476/head
2025-09-07T07:36:04.0358894Z  * [new branch]              gh/malfet/476/orig          -> origin/gh/malfet/476/orig
2025-09-07T07:36:04.0360508Z  * [new branch]              gh/malfet/477/base          -> origin/gh/malfet/477/base
2025-09-07T07:36:04.0361724Z  * [new branch]              gh/malfet/477/head          -> origin/gh/malfet/477/head
2025-09-07T07:36:04.0363194Z  * [new branch]              gh/malfet/477/orig          -> origin/gh/malfet/477/orig
2025-09-07T07:36:04.0364776Z  * [new branch]              gh/malfet/478/base          -> origin/gh/malfet/478/base
2025-09-07T07:36:04.0365941Z  * [new branch]              gh/malfet/478/head          -> origin/gh/malfet/478/head
2025-09-07T07:36:04.0367133Z  * [new branch]              gh/malfet/478/orig          -> origin/gh/malfet/478/orig
2025-09-07T07:36:04.0368785Z  * [new branch]              gh/malfet/479/base          -> origin/gh/malfet/479/base
2025-09-07T07:36:04.0370012Z  * [new branch]              gh/malfet/479/head          -> origin/gh/malfet/479/head
2025-09-07T07:36:04.0371215Z  * [new branch]              gh/malfet/479/orig          -> origin/gh/malfet/479/orig
2025-09-07T07:36:04.0373325Z  * [new branch]              gh/malfet/480/base          -> origin/gh/malfet/480/base
2025-09-07T07:36:04.0374548Z  * [new branch]              gh/malfet/480/head          -> origin/gh/malfet/480/head
2025-09-07T07:36:04.0375746Z  * [new branch]              gh/malfet/480/orig          -> origin/gh/malfet/480/orig
2025-09-07T07:36:04.0377496Z  * [new branch]              gh/malfet/481/base          -> origin/gh/malfet/481/base
2025-09-07T07:36:04.0378779Z  * [new branch]              gh/malfet/481/head          -> origin/gh/malfet/481/head
2025-09-07T07:36:04.0379929Z  * [new branch]              gh/malfet/481/orig          -> origin/gh/malfet/481/orig
2025-09-07T07:36:04.0381542Z  * [new branch]              gh/malfet/482/base          -> origin/gh/malfet/482/base
2025-09-07T07:36:04.0383202Z  * [new branch]              gh/malfet/482/head          -> origin/gh/malfet/482/head
2025-09-07T07:36:04.0384360Z  * [new branch]              gh/malfet/482/orig          -> origin/gh/malfet/482/orig
2025-09-07T07:36:04.0386500Z  * [new branch]              gh/malfet/483/base          -> origin/gh/malfet/483/base
2025-09-07T07:36:04.0387700Z  * [new branch]              gh/malfet/483/head          -> origin/gh/malfet/483/head
2025-09-07T07:36:04.0388930Z  * [new branch]              gh/malfet/483/orig          -> origin/gh/malfet/483/orig
2025-09-07T07:36:04.0390695Z  * [new branch]              gh/malfet/484/base          -> origin/gh/malfet/484/base
2025-09-07T07:36:04.0391865Z  * [new branch]              gh/malfet/484/head          -> origin/gh/malfet/484/head
2025-09-07T07:36:04.0393093Z  * [new branch]              gh/malfet/484/orig          -> origin/gh/malfet/484/orig
2025-09-07T07:36:04.0394919Z  * [new branch]              gh/malfet/485/base          -> origin/gh/malfet/485/base
2025-09-07T07:36:04.0396108Z  * [new branch]              gh/malfet/485/head          -> origin/gh/malfet/485/head
2025-09-07T07:36:04.0397328Z  * [new branch]              gh/malfet/485/orig          -> origin/gh/malfet/485/orig
2025-09-07T07:36:04.0399221Z  * [new branch]              gh/malfet/486/base          -> origin/gh/malfet/486/base
2025-09-07T07:36:04.0400402Z  * [new branch]              gh/malfet/486/head          -> origin/gh/malfet/486/head
2025-09-07T07:36:04.0401581Z  * [new branch]              gh/malfet/486/orig          -> origin/gh/malfet/486/orig
2025-09-07T07:36:04.0403512Z  * [new branch]              gh/malfet/487/base          -> origin/gh/malfet/487/base
2025-09-07T07:36:04.0408574Z  * [new branch]              gh/malfet/487/head          -> origin/gh/malfet/487/head
2025-09-07T07:36:04.0409032Z  * [new branch]              gh/malfet/487/orig          -> origin/gh/malfet/487/orig
2025-09-07T07:36:04.0411650Z  * [new branch]              gh/malfet/488/base          -> origin/gh/malfet/488/base
2025-09-07T07:36:04.0412898Z  * [new branch]              gh/malfet/488/head          -> origin/gh/malfet/488/head
2025-09-07T07:36:04.0414158Z  * [new branch]              gh/malfet/488/orig          -> origin/gh/malfet/488/orig
2025-09-07T07:36:04.0416088Z  * [new branch]              gh/malfet/489/base          -> origin/gh/malfet/489/base
2025-09-07T07:36:04.0417497Z  * [new branch]              gh/malfet/489/head          -> origin/gh/malfet/489/head
2025-09-07T07:36:04.0418863Z  * [new branch]              gh/malfet/489/orig          -> origin/gh/malfet/489/orig
2025-09-07T07:36:04.0420983Z  * [new branch]              gh/malfet/490/base          -> origin/gh/malfet/490/base
2025-09-07T07:36:04.0422296Z  * [new branch]              gh/malfet/490/head          -> origin/gh/malfet/490/head
2025-09-07T07:36:04.0423717Z  * [new branch]              gh/malfet/490/orig          -> origin/gh/malfet/490/orig
2025-09-07T07:36:04.0425608Z  * [new branch]              gh/malfet/491/base          -> origin/gh/malfet/491/base
2025-09-07T07:36:04.0426846Z  * [new branch]              gh/malfet/491/head          -> origin/gh/malfet/491/head
2025-09-07T07:36:04.0428122Z  * [new branch]              gh/malfet/491/orig          -> origin/gh/malfet/491/orig
2025-09-07T07:36:04.0429843Z  * [new branch]              gh/malfet/492/base          -> origin/gh/malfet/492/base
2025-09-07T07:36:04.0431199Z  * [new branch]              gh/malfet/492/head          -> origin/gh/malfet/492/head
2025-09-07T07:36:04.0432430Z  * [new branch]              gh/malfet/492/orig          -> origin/gh/malfet/492/orig
2025-09-07T07:36:04.0434415Z  * [new branch]              gh/malfet/493/base          -> origin/gh/malfet/493/base
2025-09-07T07:36:04.0435446Z  * [new branch]              gh/malfet/493/head          -> origin/gh/malfet/493/head
2025-09-07T07:36:04.0436682Z  * [new branch]              gh/malfet/493/orig          -> origin/gh/malfet/493/orig
2025-09-07T07:36:04.0438612Z  * [new branch]              gh/malfet/494/base          -> origin/gh/malfet/494/base
2025-09-07T07:36:04.0440011Z  * [new branch]              gh/malfet/494/head          -> origin/gh/malfet/494/head
2025-09-07T07:36:04.0441237Z  * [new branch]              gh/malfet/494/orig          -> origin/gh/malfet/494/orig
2025-09-07T07:36:04.0442980Z  * [new branch]              gh/malfet/495/base          -> origin/gh/malfet/495/base
2025-09-07T07:36:04.0444238Z  * [new branch]              gh/malfet/495/head          -> origin/gh/malfet/495/head
2025-09-07T07:36:04.0445451Z  * [new branch]              gh/malfet/495/orig          -> origin/gh/malfet/495/orig
2025-09-07T07:36:04.0447297Z  * [new branch]              gh/malfet/496/base          -> origin/gh/malfet/496/base
2025-09-07T07:36:04.0448500Z  * [new branch]              gh/malfet/496/head          -> origin/gh/malfet/496/head
2025-09-07T07:36:04.0449764Z  * [new branch]              gh/malfet/496/orig          -> origin/gh/malfet/496/orig
2025-09-07T07:36:04.0451543Z  * [new branch]              gh/malfet/497/base          -> origin/gh/malfet/497/base
2025-09-07T07:36:04.0452749Z  * [new branch]              gh/malfet/497/head          -> origin/gh/malfet/497/head
2025-09-07T07:36:04.0454017Z  * [new branch]              gh/malfet/497/orig          -> origin/gh/malfet/497/orig
2025-09-07T07:36:04.0455800Z  * [new branch]              gh/malfet/498/base          -> origin/gh/malfet/498/base
2025-09-07T07:36:04.0456982Z  * [new branch]              gh/malfet/498/head          -> origin/gh/malfet/498/head
2025-09-07T07:36:04.0458247Z  * [new branch]              gh/malfet/498/orig          -> origin/gh/malfet/498/orig
2025-09-07T07:36:04.0459969Z  * [new branch]              gh/malfet/499/base          -> origin/gh/malfet/499/base
2025-09-07T07:36:04.0461211Z  * [new branch]              gh/malfet/499/head          -> origin/gh/malfet/499/head
2025-09-07T07:36:04.0462542Z  * [new branch]              gh/malfet/499/orig          -> origin/gh/malfet/499/orig
2025-09-07T07:36:04.0464352Z  * [new branch]              gh/malfet/500/base          -> origin/gh/malfet/500/base
2025-09-07T07:36:04.0465546Z  * [new branch]              gh/malfet/500/head          -> origin/gh/malfet/500/head
2025-09-07T07:36:04.0466822Z  * [new branch]              gh/malfet/500/orig          -> origin/gh/malfet/500/orig
2025-09-07T07:36:04.0469190Z  * [new branch]              gh/malfet/501/base          -> origin/gh/malfet/501/base
2025-09-07T07:36:04.0470370Z  * [new branch]              gh/malfet/501/head          -> origin/gh/malfet/501/head
2025-09-07T07:36:04.0471553Z  * [new branch]              gh/malfet/501/orig          -> origin/gh/malfet/501/orig
2025-09-07T07:36:04.0473441Z  * [new branch]              gh/malfet/502/base          -> origin/gh/malfet/502/base
2025-09-07T07:36:04.0474617Z  * [new branch]              gh/malfet/502/head          -> origin/gh/malfet/502/head
2025-09-07T07:36:04.0475850Z  * [new branch]              gh/malfet/502/orig          -> origin/gh/malfet/502/orig
2025-09-07T07:36:04.0477847Z  * [new branch]              gh/malfet/503/base          -> origin/gh/malfet/503/base
2025-09-07T07:36:04.0479078Z  * [new branch]              gh/malfet/503/head          -> origin/gh/malfet/503/head
2025-09-07T07:36:04.0480253Z  * [new branch]              gh/malfet/503/orig          -> origin/gh/malfet/503/orig
2025-09-07T07:36:04.0482059Z  * [new branch]              gh/malfet/504/base          -> origin/gh/malfet/504/base
2025-09-07T07:36:04.0483229Z  * [new branch]              gh/malfet/504/head          -> origin/gh/malfet/504/head
2025-09-07T07:36:04.0484537Z  * [new branch]              gh/malfet/504/orig          -> origin/gh/malfet/504/orig
2025-09-07T07:36:04.0486302Z  * [new branch]              gh/malfet/505/base          -> origin/gh/malfet/505/base
2025-09-07T07:36:04.0487496Z  * [new branch]              gh/malfet/505/head          -> origin/gh/malfet/505/head
2025-09-07T07:36:04.0488684Z  * [new branch]              gh/malfet/505/orig          -> origin/gh/malfet/505/orig
2025-09-07T07:36:04.0490554Z  * [new branch]              gh/malfet/506/base          -> origin/gh/malfet/506/base
2025-09-07T07:36:04.0491678Z  * [new branch]              gh/malfet/506/head          -> origin/gh/malfet/506/head
2025-09-07T07:36:04.0492849Z  * [new branch]              gh/malfet/506/orig          -> origin/gh/malfet/506/orig
2025-09-07T07:36:04.0494702Z  * [new branch]              gh/malfet/507/base          -> origin/gh/malfet/507/base
2025-09-07T07:36:04.0495883Z  * [new branch]              gh/malfet/507/head          -> origin/gh/malfet/507/head
2025-09-07T07:36:04.0497109Z  * [new branch]              gh/malfet/507/orig          -> origin/gh/malfet/507/orig
2025-09-07T07:36:04.0499073Z  * [new branch]              gh/malfet/508/base          -> origin/gh/malfet/508/base
2025-09-07T07:36:04.0500304Z  * [new branch]              gh/malfet/508/head          -> origin/gh/malfet/508/head
2025-09-07T07:36:04.0501525Z  * [new branch]              gh/malfet/508/orig          -> origin/gh/malfet/508/orig
2025-09-07T07:36:04.0503342Z  * [new branch]              gh/malfet/509/base          -> origin/gh/malfet/509/base
2025-09-07T07:36:04.0504775Z  * [new branch]              gh/malfet/509/head          -> origin/gh/malfet/509/head
2025-09-07T07:36:04.0506052Z  * [new branch]              gh/malfet/509/orig          -> origin/gh/malfet/509/orig
2025-09-07T07:36:04.0507902Z  * [new branch]              gh/malfet/510/base          -> origin/gh/malfet/510/base
2025-09-07T07:36:04.0509097Z  * [new branch]              gh/malfet/510/head          -> origin/gh/malfet/510/head
2025-09-07T07:36:04.0510295Z  * [new branch]              gh/malfet/510/orig          -> origin/gh/malfet/510/orig
2025-09-07T07:36:04.0512576Z  * [new branch]              gh/malfet/511/base          -> origin/gh/malfet/511/base
2025-09-07T07:36:04.0514051Z  * [new branch]              gh/malfet/511/head          -> origin/gh/malfet/511/head
2025-09-07T07:36:04.0515244Z  * [new branch]              gh/malfet/511/orig          -> origin/gh/malfet/511/orig
2025-09-07T07:36:04.0517016Z  * [new branch]              gh/malfet/512/base          -> origin/gh/malfet/512/base
2025-09-07T07:36:04.0518243Z  * [new branch]              gh/malfet/512/head          -> origin/gh/malfet/512/head
2025-09-07T07:36:04.0519502Z  * [new branch]              gh/malfet/512/orig          -> origin/gh/malfet/512/orig
2025-09-07T07:36:04.0521317Z  * [new branch]              gh/malfet/513/base          -> origin/gh/malfet/513/base
2025-09-07T07:36:04.0522502Z  * [new branch]              gh/malfet/513/head          -> origin/gh/malfet/513/head
2025-09-07T07:36:04.0523648Z  * [new branch]              gh/malfet/513/orig          -> origin/gh/malfet/513/orig
2025-09-07T07:36:04.0525604Z  * [new branch]              gh/malfet/64/base           -> origin/gh/malfet/64/base
2025-09-07T07:36:04.0526857Z  * [new branch]              gh/malfet/64/head           -> origin/gh/malfet/64/head
2025-09-07T07:36:04.0529353Z  * [new branch]              gh/manuelcandales/10/base   -> origin/gh/manuelcandales/10/base
2025-09-07T07:36:04.0530474Z  * [new branch]              gh/manuelcandales/10/head   -> origin/gh/manuelcandales/10/head
2025-09-07T07:36:04.0531628Z  * [new branch]              gh/manuelcandales/10/orig   -> origin/gh/manuelcandales/10/orig
2025-09-07T07:36:04.0533345Z  * [new branch]              gh/manuelcandales/11/base   -> origin/gh/manuelcandales/11/base
2025-09-07T07:36:04.0534555Z  * [new branch]              gh/manuelcandales/11/head   -> origin/gh/manuelcandales/11/head
2025-09-07T07:36:04.0535805Z  * [new branch]              gh/manuelcandales/11/orig   -> origin/gh/manuelcandales/11/orig
2025-09-07T07:36:04.0537518Z  * [new branch]              gh/manuelcandales/9/base    -> origin/gh/manuelcandales/9/base
2025-09-07T07:36:04.0538786Z  * [new branch]              gh/manuelcandales/9/head    -> origin/gh/manuelcandales/9/head
2025-09-07T07:36:04.0540042Z  * [new branch]              gh/manuelcandales/9/orig    -> origin/gh/manuelcandales/9/orig
2025-09-07T07:36:04.0542477Z  * [new branch]              gh/markkm/1/base            -> origin/gh/markkm/1/base
2025-09-07T07:36:04.0544480Z  * [new branch]              gh/masnesral/204/base       -> origin/gh/masnesral/204/base
2025-09-07T07:36:04.0546205Z  * [new branch]              gh/masnesral/204/head       -> origin/gh/masnesral/204/head
2025-09-07T07:36:04.0547129Z  * [new branch]              gh/masnesral/204/orig       -> origin/gh/masnesral/204/orig
2025-09-07T07:36:04.0548957Z  * [new branch]              gh/masnesral/235/base       -> origin/gh/masnesral/235/base
2025-09-07T07:36:04.0550169Z  * [new branch]              gh/masnesral/235/head       -> origin/gh/masnesral/235/head
2025-09-07T07:36:04.0551591Z  * [new branch]              gh/masnesral/235/orig       -> origin/gh/masnesral/235/orig
2025-09-07T07:36:04.0553362Z  * [new branch]              gh/masnesral/34/base        -> origin/gh/masnesral/34/base
2025-09-07T07:36:04.0555540Z  * [new branch]              gh/mhorowitz/0/base         -> origin/gh/mhorowitz/0/base
2025-09-07T07:36:04.0556725Z  * [new branch]              gh/mhorowitz/0/head         -> origin/gh/mhorowitz/0/head
2025-09-07T07:36:04.0558320Z  * [new branch]              gh/mhorowitz/1/base         -> origin/gh/mhorowitz/1/base
2025-09-07T07:36:04.0559490Z  * [new branch]              gh/mhorowitz/1/head         -> origin/gh/mhorowitz/1/head
2025-09-07T07:36:04.0561635Z  * [new branch]              gh/mhorowitz/2/base         -> origin/gh/mhorowitz/2/base
2025-09-07T07:36:04.0562312Z  * [new branch]              gh/mhorowitz/2/head         -> origin/gh/mhorowitz/2/head
2025-09-07T07:36:04.0563948Z  * [new branch]              gh/mhorowitz/3/base         -> origin/gh/mhorowitz/3/base
2025-09-07T07:36:04.0565152Z  * [new branch]              gh/mhorowitz/3/head         -> origin/gh/mhorowitz/3/head
2025-09-07T07:36:04.0566793Z  * [new branch]              gh/mhorowitz/4/base         -> origin/gh/mhorowitz/4/base
2025-09-07T07:36:04.0567977Z  * [new branch]              gh/mhorowitz/4/head         -> origin/gh/mhorowitz/4/head
2025-09-07T07:36:04.0569558Z  * [new branch]              gh/mhorowitz/5/base         -> origin/gh/mhorowitz/5/base
2025-09-07T07:36:04.0570687Z  * [new branch]              gh/mhorowitz/5/head         -> origin/gh/mhorowitz/5/head
2025-09-07T07:36:04.0572249Z  * [new branch]              gh/mhorowitz/6/base         -> origin/gh/mhorowitz/6/base
2025-09-07T07:36:04.0573408Z  * [new branch]              gh/mhorowitz/6/head         -> origin/gh/mhorowitz/6/head
2025-09-07T07:36:04.0575585Z  * [new branch]              gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base
2025-09-07T07:36:04.0576784Z  * [new branch]              gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head
2025-09-07T07:36:04.0578490Z  * [new branch]              gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base
2025-09-07T07:36:04.0579709Z  * [new branch]              gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head
2025-09-07T07:36:04.0581424Z  * [new branch]              gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base
2025-09-07T07:36:04.0582568Z  * [new branch]              gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head
2025-09-07T07:36:04.0584253Z  * [new branch]              gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base
2025-09-07T07:36:04.0585365Z  * [new branch]              gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head
2025-09-07T07:36:04.0587017Z  * [new branch]              gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base
2025-09-07T07:36:04.0588393Z  * [new branch]              gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head
2025-09-07T07:36:04.0590197Z  * [new branch]              gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base
2025-09-07T07:36:04.0591428Z  * [new branch]              gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head
2025-09-07T07:36:04.0592672Z  * [new branch]              gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig
2025-09-07T07:36:04.0594379Z  * [new branch]              gh/mikaylagawarecki/320/base -> origin/gh/mikaylagawarecki/320/base
2025-09-07T07:36:04.0595561Z  * [new branch]              gh/mikaylagawarecki/320/head -> origin/gh/mikaylagawarecki/320/head
2025-09-07T07:36:04.0596806Z  * [new branch]              gh/mikaylagawarecki/320/orig -> origin/gh/mikaylagawarecki/320/orig
2025-09-07T07:36:04.0598524Z  * [new branch]              gh/mikaylagawarecki/329/base -> origin/gh/mikaylagawarecki/329/base
2025-09-07T07:36:04.0599712Z  * [new branch]              gh/mikaylagawarecki/329/head -> origin/gh/mikaylagawarecki/329/head
2025-09-07T07:36:04.0600885Z  * [new branch]              gh/mikaylagawarecki/329/orig -> origin/gh/mikaylagawarecki/329/orig
2025-09-07T07:36:04.0602584Z  * [new branch]              gh/mikaylagawarecki/330/base -> origin/gh/mikaylagawarecki/330/base
2025-09-07T07:36:04.0603996Z  * [new branch]              gh/mikaylagawarecki/330/head -> origin/gh/mikaylagawarecki/330/head
2025-09-07T07:36:04.0605407Z  * [new branch]              gh/mikaylagawarecki/330/orig -> origin/gh/mikaylagawarecki/330/orig
2025-09-07T07:36:04.0607135Z  * [new branch]              gh/mikaylagawarecki/331/base -> origin/gh/mikaylagawarecki/331/base
2025-09-07T07:36:04.0608339Z  * [new branch]              gh/mikaylagawarecki/331/head -> origin/gh/mikaylagawarecki/331/head
2025-09-07T07:36:04.0609506Z  * [new branch]              gh/mikaylagawarecki/331/orig -> origin/gh/mikaylagawarecki/331/orig
2025-09-07T07:36:04.0611415Z  * [new branch]              gh/mikaylagawarecki/332/base -> origin/gh/mikaylagawarecki/332/base
2025-09-07T07:36:04.0612573Z  * [new branch]              gh/mikaylagawarecki/332/head -> origin/gh/mikaylagawarecki/332/head
2025-09-07T07:36:04.0613848Z  * [new branch]              gh/mikaylagawarecki/332/orig -> origin/gh/mikaylagawarecki/332/orig
2025-09-07T07:36:04.0616481Z  * [new branch]              gh/mikaylagawarecki/334/base -> origin/gh/mikaylagawarecki/334/base
2025-09-07T07:36:04.0617637Z  * [new branch]              gh/mikaylagawarecki/334/head -> origin/gh/mikaylagawarecki/334/head
2025-09-07T07:36:04.0618861Z  * [new branch]              gh/mikaylagawarecki/334/orig -> origin/gh/mikaylagawarecki/334/orig
2025-09-07T07:36:04.0620596Z  * [new branch]              gh/mikaylagawarecki/335/base -> origin/gh/mikaylagawarecki/335/base
2025-09-07T07:36:04.0621840Z  * [new branch]              gh/mikaylagawarecki/335/head -> origin/gh/mikaylagawarecki/335/head
2025-09-07T07:36:04.0623012Z  * [new branch]              gh/mikaylagawarecki/335/orig -> origin/gh/mikaylagawarecki/335/orig
2025-09-07T07:36:04.0624938Z  * [new branch]              gh/mikaylagawarecki/336/base -> origin/gh/mikaylagawarecki/336/base
2025-09-07T07:36:04.0626183Z  * [new branch]              gh/mikaylagawarecki/336/head -> origin/gh/mikaylagawarecki/336/head
2025-09-07T07:36:04.0627351Z  * [new branch]              gh/mikaylagawarecki/336/orig -> origin/gh/mikaylagawarecki/336/orig
2025-09-07T07:36:04.0628953Z  * [new branch]              gh/mikaylagawarecki/337/base -> origin/gh/mikaylagawarecki/337/base
2025-09-07T07:36:04.0630417Z  * [new branch]              gh/mikaylagawarecki/337/head -> origin/gh/mikaylagawarecki/337/head
2025-09-07T07:36:04.0631703Z  * [new branch]              gh/mikaylagawarecki/337/orig -> origin/gh/mikaylagawarecki/337/orig
2025-09-07T07:36:04.0633617Z  * [new branch]              gh/mikaylagawarecki/338/base -> origin/gh/mikaylagawarecki/338/base
2025-09-07T07:36:04.0634847Z  * [new branch]              gh/mikaylagawarecki/338/head -> origin/gh/mikaylagawarecki/338/head
2025-09-07T07:36:04.0636038Z  * [new branch]              gh/mikaylagawarecki/338/orig -> origin/gh/mikaylagawarecki/338/orig
2025-09-07T07:36:04.0638161Z  * [new branch]              gh/mikaylagawarecki/339/base -> origin/gh/mikaylagawarecki/339/base
2025-09-07T07:36:04.0639369Z  * [new branch]              gh/mikaylagawarecki/339/head -> origin/gh/mikaylagawarecki/339/head
2025-09-07T07:36:04.0640957Z  * [new branch]              gh/mikaylagawarecki/339/orig -> origin/gh/mikaylagawarecki/339/orig
2025-09-07T07:36:04.0642761Z  * [new branch]              gh/mlazos/1/base            -> origin/gh/mlazos/1/base
2025-09-07T07:36:04.0643991Z  * [new branch]              gh/mlazos/1/head            -> origin/gh/mlazos/1/head
2025-09-07T07:36:04.0645175Z  * [new branch]              gh/mlazos/1/orig            -> origin/gh/mlazos/1/orig
2025-09-07T07:36:04.0646937Z  * [new branch]              gh/mlazos/12/base           -> origin/gh/mlazos/12/base
2025-09-07T07:36:04.0648229Z  * [new branch]              gh/mlazos/12/head           -> origin/gh/mlazos/12/head
2025-09-07T07:36:04.0649438Z  * [new branch]              gh/mlazos/12/orig           -> origin/gh/mlazos/12/orig
2025-09-07T07:36:04.0651250Z  * [new branch]              gh/mlazos/13/base           -> origin/gh/mlazos/13/base
2025-09-07T07:36:04.0652467Z  * [new branch]              gh/mlazos/13/head           -> origin/gh/mlazos/13/head
2025-09-07T07:36:04.0653673Z  * [new branch]              gh/mlazos/13/orig           -> origin/gh/mlazos/13/orig
2025-09-07T07:36:04.0655386Z  * [new branch]              gh/mlazos/14/base           -> origin/gh/mlazos/14/base
2025-09-07T07:36:04.0656580Z  * [new branch]              gh/mlazos/14/head           -> origin/gh/mlazos/14/head
2025-09-07T07:36:04.0657868Z  * [new branch]              gh/mlazos/14/orig           -> origin/gh/mlazos/14/orig
2025-09-07T07:36:04.0659656Z  * [new branch]              gh/mlazos/15/base           -> origin/gh/mlazos/15/base
2025-09-07T07:36:04.0661017Z  * [new branch]              gh/mlazos/15/head           -> origin/gh/mlazos/15/head
2025-09-07T07:36:04.0662187Z  * [new branch]              gh/mlazos/15/orig           -> origin/gh/mlazos/15/orig
2025-09-07T07:36:04.0663987Z  * [new branch]              gh/mlazos/16/base           -> origin/gh/mlazos/16/base
2025-09-07T07:36:04.0665293Z  * [new branch]              gh/mlazos/16/head           -> origin/gh/mlazos/16/head
2025-09-07T07:36:04.0666464Z  * [new branch]              gh/mlazos/16/orig           -> origin/gh/mlazos/16/orig
2025-09-07T07:36:04.0668114Z  * [new branch]              gh/mlazos/17/base           -> origin/gh/mlazos/17/base
2025-09-07T07:36:04.0669245Z  * [new branch]              gh/mlazos/17/head           -> origin/gh/mlazos/17/head
2025-09-07T07:36:04.0670440Z  * [new branch]              gh/mlazos/17/orig           -> origin/gh/mlazos/17/orig
2025-09-07T07:36:04.0672306Z  * [new branch]              gh/mlazos/2/base            -> origin/gh/mlazos/2/base
2025-09-07T07:36:04.0673408Z  * [new branch]              gh/mlazos/2/head            -> origin/gh/mlazos/2/head
2025-09-07T07:36:04.0674609Z  * [new branch]              gh/mlazos/2/orig            -> origin/gh/mlazos/2/orig
2025-09-07T07:36:04.0676946Z  * [new branch]              gh/mlazos/3/base            -> origin/gh/mlazos/3/base
2025-09-07T07:36:04.0678090Z  * [new branch]              gh/mlazos/3/head            -> origin/gh/mlazos/3/head
2025-09-07T07:36:04.0679260Z  * [new branch]              gh/mlazos/3/orig            -> origin/gh/mlazos/3/orig
2025-09-07T07:36:04.0681503Z  * [new branch]              gh/mrmiywj/1/base           -> origin/gh/mrmiywj/1/base
2025-09-07T07:36:04.0682864Z  * [new branch]              gh/mrmiywj/1/head           -> origin/gh/mrmiywj/1/head
2025-09-07T07:36:04.0685052Z  * [new branch]              gh/muchulee8/62/base        -> origin/gh/muchulee8/62/base
2025-09-07T07:36:04.0686311Z  * [new branch]              gh/muchulee8/62/head        -> origin/gh/muchulee8/62/head
2025-09-07T07:36:04.0687512Z  * [new branch]              gh/muchulee8/62/orig        -> origin/gh/muchulee8/62/orig
2025-09-07T07:36:04.0689247Z  * [new branch]              gh/muchulee8/63/base        -> origin/gh/muchulee8/63/base
2025-09-07T07:36:04.0690417Z  * [new branch]              gh/muchulee8/63/head        -> origin/gh/muchulee8/63/head
2025-09-07T07:36:04.0691686Z  * [new branch]              gh/muchulee8/63/orig        -> origin/gh/muchulee8/63/orig
2025-09-07T07:36:04.0693463Z  * [new branch]              gh/muchulee8/64/base        -> origin/gh/muchulee8/64/base
2025-09-07T07:36:04.0694615Z  * [new branch]              gh/muchulee8/64/head        -> origin/gh/muchulee8/64/head
2025-09-07T07:36:04.0695861Z  * [new branch]              gh/muchulee8/64/orig        -> origin/gh/muchulee8/64/orig
2025-09-07T07:36:04.0697751Z  * [new branch]              gh/muchulee8/65/base        -> origin/gh/muchulee8/65/base
2025-09-07T07:36:04.0699198Z  * [new branch]              gh/muchulee8/65/head        -> origin/gh/muchulee8/65/head
2025-09-07T07:36:04.0700898Z  * [new branch]              gh/muchulee8/65/orig        -> origin/gh/muchulee8/65/orig
2025-09-07T07:36:04.0703241Z  * [new branch]              gh/naveenthangudu/1/base    -> origin/gh/naveenthangudu/1/base
2025-09-07T07:36:04.0704529Z  * [new branch]              gh/naveenthangudu/1/head    -> origin/gh/naveenthangudu/1/head
2025-09-07T07:36:04.0705802Z  * [new branch]              gh/naveenthangudu/1/orig    -> origin/gh/naveenthangudu/1/orig
2025-09-07T07:36:04.0707483Z  * [new branch]              gh/naveenthangudu/2/base    -> origin/gh/naveenthangudu/2/base
2025-09-07T07:36:04.0708663Z  * [new branch]              gh/naveenthangudu/2/head    -> origin/gh/naveenthangudu/2/head
2025-09-07T07:36:04.0709893Z  * [new branch]              gh/naveenthangudu/2/orig    -> origin/gh/naveenthangudu/2/orig
2025-09-07T07:36:04.0711560Z  * [new branch]              gh/naveenthangudu/3/base    -> origin/gh/naveenthangudu/3/base
2025-09-07T07:36:04.0712835Z  * [new branch]              gh/naveenthangudu/3/head    -> origin/gh/naveenthangudu/3/head
2025-09-07T07:36:04.0714045Z  * [new branch]              gh/naveenthangudu/3/orig    -> origin/gh/naveenthangudu/3/orig
2025-09-07T07:36:04.0715742Z  * [new branch]              gh/naveenthangudu/4/base    -> origin/gh/naveenthangudu/4/base
2025-09-07T07:36:04.0716962Z  * [new branch]              gh/naveenthangudu/4/head    -> origin/gh/naveenthangudu/4/head
2025-09-07T07:36:04.0718202Z  * [new branch]              gh/naveenthangudu/4/orig    -> origin/gh/naveenthangudu/4/orig
2025-09-07T07:36:04.0719905Z  * [new branch]              gh/naveenthangudu/5/base    -> origin/gh/naveenthangudu/5/base
2025-09-07T07:36:04.0721061Z  * [new branch]              gh/naveenthangudu/5/head    -> origin/gh/naveenthangudu/5/head
2025-09-07T07:36:04.0722326Z  * [new branch]              gh/naveenthangudu/5/orig    -> origin/gh/naveenthangudu/5/orig
2025-09-07T07:36:04.0723987Z  * [new branch]              gh/naveenthangudu/6/base    -> origin/gh/naveenthangudu/6/base
2025-09-07T07:36:04.0725178Z  * [new branch]              gh/naveenthangudu/6/head    -> origin/gh/naveenthangudu/6/head
2025-09-07T07:36:04.0726311Z  * [new branch]              gh/naveenthangudu/6/orig    -> origin/gh/naveenthangudu/6/orig
2025-09-07T07:36:04.0728401Z  * [new branch]              gh/oulgen/35/base           -> origin/gh/oulgen/35/base
2025-09-07T07:36:04.0729633Z  * [new branch]              gh/oulgen/35/head           -> origin/gh/oulgen/35/head
2025-09-07T07:36:04.0730845Z  * [new branch]              gh/oulgen/35/orig           -> origin/gh/oulgen/35/orig
2025-09-07T07:36:04.0732546Z  * [new branch]              gh/oulgen/48/base           -> origin/gh/oulgen/48/base
2025-09-07T07:36:04.0733700Z  * [new branch]              gh/oulgen/48/head           -> origin/gh/oulgen/48/head
2025-09-07T07:36:04.0735088Z  * [new branch]              gh/oulgen/48/orig           -> origin/gh/oulgen/48/orig
2025-09-07T07:36:04.0737200Z  * [new branch]              gh/oulgen/49/base           -> origin/gh/oulgen/49/base
2025-09-07T07:36:04.0738439Z  * [new branch]              gh/oulgen/49/head           -> origin/gh/oulgen/49/head
2025-09-07T07:36:04.0739649Z  * [new branch]              gh/oulgen/49/orig           -> origin/gh/oulgen/49/orig
2025-09-07T07:36:04.0741963Z  * [new branch]              gh/pearu/108/base           -> origin/gh/pearu/108/base
2025-09-07T07:36:04.0743278Z  * [new branch]              gh/pearu/108/head           -> origin/gh/pearu/108/head
2025-09-07T07:36:04.0744556Z  * [new branch]              gh/pearu/108/orig           -> origin/gh/pearu/108/orig
2025-09-07T07:36:04.0746671Z  * [new branch]              gh/pearu/109/base           -> origin/gh/pearu/109/base
2025-09-07T07:36:04.0747924Z  * [new branch]              gh/pearu/109/head           -> origin/gh/pearu/109/head
2025-09-07T07:36:04.0749106Z  * [new branch]              gh/pearu/109/orig           -> origin/gh/pearu/109/orig
2025-09-07T07:36:04.0750836Z  * [new branch]              gh/pearu/110/base           -> origin/gh/pearu/110/base
2025-09-07T07:36:04.0752044Z  * [new branch]              gh/pearu/110/head           -> origin/gh/pearu/110/head
2025-09-07T07:36:04.0753189Z  * [new branch]              gh/pearu/110/orig           -> origin/gh/pearu/110/orig
2025-09-07T07:36:04.0755081Z  * [new branch]              gh/pearu/111/base           -> origin/gh/pearu/111/base
2025-09-07T07:36:04.0756254Z  * [new branch]              gh/pearu/111/head           -> origin/gh/pearu/111/head
2025-09-07T07:36:04.0757464Z  * [new branch]              gh/pearu/111/orig           -> origin/gh/pearu/111/orig
2025-09-07T07:36:04.0759076Z  * [new branch]              gh/pearu/112/base           -> origin/gh/pearu/112/base
2025-09-07T07:36:04.0760270Z  * [new branch]              gh/pearu/112/head           -> origin/gh/pearu/112/head
2025-09-07T07:36:04.0761525Z  * [new branch]              gh/pearu/112/orig           -> origin/gh/pearu/112/orig
2025-09-07T07:36:04.0763169Z  * [new branch]              gh/pearu/113/base           -> origin/gh/pearu/113/base
2025-09-07T07:36:04.0764448Z  * [new branch]              gh/pearu/113/head           -> origin/gh/pearu/113/head
2025-09-07T07:36:04.0765612Z  * [new branch]              gh/pearu/113/orig           -> origin/gh/pearu/113/orig
2025-09-07T07:36:04.0767325Z  * [new branch]              gh/pearu/114/base           -> origin/gh/pearu/114/base
2025-09-07T07:36:04.0768474Z  * [new branch]              gh/pearu/114/head           -> origin/gh/pearu/114/head
2025-09-07T07:36:04.0769729Z  * [new branch]              gh/pearu/114/orig           -> origin/gh/pearu/114/orig
2025-09-07T07:36:04.0771648Z  * [new branch]              gh/pearu/115/base           -> origin/gh/pearu/115/base
2025-09-07T07:36:04.0772845Z  * [new branch]              gh/pearu/115/head           -> origin/gh/pearu/115/head
2025-09-07T07:36:04.0774021Z  * [new branch]              gh/pearu/115/orig           -> origin/gh/pearu/115/orig
2025-09-07T07:36:04.0775741Z  * [new branch]              gh/pearu/116/base           -> origin/gh/pearu/116/base
2025-09-07T07:36:04.0776877Z  * [new branch]              gh/pearu/116/head           -> origin/gh/pearu/116/head
2025-09-07T07:36:04.0778146Z  * [new branch]              gh/pearu/116/orig           -> origin/gh/pearu/116/orig
2025-09-07T07:36:04.0779851Z  * [new branch]              gh/pearu/117/base           -> origin/gh/pearu/117/base
2025-09-07T07:36:04.0781030Z  * [new branch]              gh/pearu/117/head           -> origin/gh/pearu/117/head
2025-09-07T07:36:04.0782149Z  * [new branch]              gh/pearu/117/orig           -> origin/gh/pearu/117/orig
2025-09-07T07:36:04.0784194Z  * [new branch]              gh/pearu/56/base            -> origin/gh/pearu/56/base
2025-09-07T07:36:04.0785517Z  * [new branch]              gh/pearu/56/head            -> origin/gh/pearu/56/head
2025-09-07T07:36:04.0786744Z  * [new branch]              gh/pearu/56/orig            -> origin/gh/pearu/56/orig
2025-09-07T07:36:04.0788601Z  * [new branch]              gh/pearu/97/base            -> origin/gh/pearu/97/base
2025-09-07T07:36:04.0789806Z  * [new branch]              gh/pearu/97/head            -> origin/gh/pearu/97/head
2025-09-07T07:36:04.0791073Z  * [new branch]              gh/pearu/97/orig            -> origin/gh/pearu/97/orig
2025-09-07T07:36:04.0793237Z  * [new branch]              gh/qqaatw/29/base           -> origin/gh/qqaatw/29/base
2025-09-07T07:36:04.0794954Z  * [new branch]              gh/qqaatw/29/head           -> origin/gh/qqaatw/29/head
2025-09-07T07:36:04.0796114Z  * [new branch]              gh/qqaatw/29/orig           -> origin/gh/qqaatw/29/orig
2025-09-07T07:36:04.0797921Z  * [new branch]              gh/raymo/refresh-script     -> origin/gh/raymo/refresh-script
2025-09-07T07:36:04.0800067Z  * [new branch]              gh/rec/141/base             -> origin/gh/rec/141/base
2025-09-07T07:36:04.0801288Z  * [new branch]              gh/rec/141/head             -> origin/gh/rec/141/head
2025-09-07T07:36:04.0802982Z  * [new branch]              gh/rec/153/base             -> origin/gh/rec/153/base
2025-09-07T07:36:04.0804405Z  * [new branch]              gh/rec/153/head             -> origin/gh/rec/153/head
2025-09-07T07:36:04.0805598Z  * [new branch]              gh/rec/153/orig             -> origin/gh/rec/153/orig
2025-09-07T07:36:04.0807279Z  * [new branch]              gh/rec/154/base             -> origin/gh/rec/154/base
2025-09-07T07:36:04.0808610Z  * [new branch]              gh/rec/154/head             -> origin/gh/rec/154/head
2025-09-07T07:36:04.0809790Z  * [new branch]              gh/rec/154/orig             -> origin/gh/rec/154/orig
2025-09-07T07:36:04.0811444Z  * [new branch]              gh/rec/156/base             -> origin/gh/rec/156/base
2025-09-07T07:36:04.0813094Z  * [new branch]              gh/rec/156/head             -> origin/gh/rec/156/head
2025-09-07T07:36:04.0814381Z  * [new branch]              gh/rec/156/orig             -> origin/gh/rec/156/orig
2025-09-07T07:36:04.0816095Z  * [new branch]              gh/rec/160/base             -> origin/gh/rec/160/base
2025-09-07T07:36:04.0817272Z  * [new branch]              gh/rec/160/head             -> origin/gh/rec/160/head
2025-09-07T07:36:04.0818516Z  * [new branch]              gh/rec/160/orig             -> origin/gh/rec/160/orig
2025-09-07T07:36:04.0820237Z  * [new branch]              gh/rec/162/base             -> origin/gh/rec/162/base
2025-09-07T07:36:04.0821424Z  * [new branch]              gh/rec/162/head             -> origin/gh/rec/162/head
2025-09-07T07:36:04.0822614Z  * [new branch]              gh/rec/162/orig             -> origin/gh/rec/162/orig
2025-09-07T07:36:04.0824268Z  * [new branch]              gh/rec/163/base             -> origin/gh/rec/163/base
2025-09-07T07:36:04.0825432Z  * [new branch]              gh/rec/163/head             -> origin/gh/rec/163/head
2025-09-07T07:36:04.0826588Z  * [new branch]              gh/rec/163/orig             -> origin/gh/rec/163/orig
2025-09-07T07:36:04.0828255Z  * [new branch]              gh/rec/164/base             -> origin/gh/rec/164/base
2025-09-07T07:36:04.0829507Z  * [new branch]              gh/rec/164/head             -> origin/gh/rec/164/head
2025-09-07T07:36:04.0830712Z  * [new branch]              gh/rec/164/orig             -> origin/gh/rec/164/orig
2025-09-07T07:36:04.0832444Z  * [new branch]              gh/rec/165/base             -> origin/gh/rec/165/base
2025-09-07T07:36:04.0833739Z  * [new branch]              gh/rec/165/head             -> origin/gh/rec/165/head
2025-09-07T07:36:04.0834899Z  * [new branch]              gh/rec/165/orig             -> origin/gh/rec/165/orig
2025-09-07T07:36:04.0836554Z  * [new branch]              gh/rec/166/base             -> origin/gh/rec/166/base
2025-09-07T07:36:04.0837768Z  * [new branch]              gh/rec/166/head             -> origin/gh/rec/166/head
2025-09-07T07:36:04.0838927Z  * [new branch]              gh/rec/166/orig             -> origin/gh/rec/166/orig
2025-09-07T07:36:04.0841105Z  * [new branch]              gh/robert-hardwick/1/base   -> origin/gh/robert-hardwick/1/base
2025-09-07T07:36:04.0842311Z  * [new branch]              gh/robert-hardwick/1/head   -> origin/gh/robert-hardwick/1/head
2025-09-07T07:36:04.0843684Z  * [new branch]              gh/robert-hardwick/1/orig   -> origin/gh/robert-hardwick/1/orig
2025-09-07T07:36:04.0845429Z  * [new branch]              gh/robert-hardwick/2/base   -> origin/gh/robert-hardwick/2/base
2025-09-07T07:36:04.0846835Z  * [new branch]              gh/robert-hardwick/2/head   -> origin/gh/robert-hardwick/2/head
2025-09-07T07:36:04.0847907Z  * [new branch]              gh/robert-hardwick/2/orig   -> origin/gh/robert-hardwick/2/orig
2025-09-07T07:36:04.0849574Z  * [new branch]              gh/robert-hardwick/3/base   -> origin/gh/robert-hardwick/3/base
2025-09-07T07:36:04.0850766Z  * [new branch]              gh/robert-hardwick/3/head   -> origin/gh/robert-hardwick/3/head
2025-09-07T07:36:04.0852086Z  * [new branch]              gh/robert-hardwick/3/orig   -> origin/gh/robert-hardwick/3/orig
2025-09-07T07:36:04.0853704Z  * [new branch]              gh/robert-hardwick/4/base   -> origin/gh/robert-hardwick/4/base
2025-09-07T07:36:04.0854861Z  * [new branch]              gh/robert-hardwick/4/head   -> origin/gh/robert-hardwick/4/head
2025-09-07T07:36:04.0856052Z  * [new branch]              gh/robert-hardwick/4/orig   -> origin/gh/robert-hardwick/4/orig
2025-09-07T07:36:04.0858235Z  * [new branch]              gh/rtimpe/1/base            -> origin/gh/rtimpe/1/base
2025-09-07T07:36:04.0859482Z  * [new branch]              gh/rtimpe/1/head            -> origin/gh/rtimpe/1/head
2025-09-07T07:36:04.0861291Z  * [new branch]              gh/rtimpe/10/base           -> origin/gh/rtimpe/10/base
2025-09-07T07:36:04.0862466Z  * [new branch]              gh/rtimpe/10/head           -> origin/gh/rtimpe/10/head
2025-09-07T07:36:04.0863761Z  * [new branch]              gh/rtimpe/10/orig           -> origin/gh/rtimpe/10/orig
2025-09-07T07:36:04.0865477Z  * [new branch]              gh/rtimpe/11/base           -> origin/gh/rtimpe/11/base
2025-09-07T07:36:04.0866672Z  * [new branch]              gh/rtimpe/11/head           -> origin/gh/rtimpe/11/head
2025-09-07T07:36:04.0867833Z  * [new branch]              gh/rtimpe/11/orig           -> origin/gh/rtimpe/11/orig
2025-09-07T07:36:04.0869499Z  * [new branch]              gh/rtimpe/12/base           -> origin/gh/rtimpe/12/base
2025-09-07T07:36:04.0870696Z  * [new branch]              gh/rtimpe/12/head           -> origin/gh/rtimpe/12/head
2025-09-07T07:36:04.0872325Z  * [new branch]              gh/rtimpe/12/orig           -> origin/gh/rtimpe/12/orig
2025-09-07T07:36:04.0873988Z  * [new branch]              gh/rtimpe/13/base           -> origin/gh/rtimpe/13/base
2025-09-07T07:36:04.0875264Z  * [new branch]              gh/rtimpe/13/head           -> origin/gh/rtimpe/13/head
2025-09-07T07:36:04.0876426Z  * [new branch]              gh/rtimpe/13/orig           -> origin/gh/rtimpe/13/orig
2025-09-07T07:36:04.0878084Z  * [new branch]              gh/rtimpe/14/base           -> origin/gh/rtimpe/14/base
2025-09-07T07:36:04.0879488Z  * [new branch]              gh/rtimpe/14/head           -> origin/gh/rtimpe/14/head
2025-09-07T07:36:04.0880707Z  * [new branch]              gh/rtimpe/14/orig           -> origin/gh/rtimpe/14/orig
2025-09-07T07:36:04.0882382Z  * [new branch]              gh/rtimpe/15/base           -> origin/gh/rtimpe/15/base
2025-09-07T07:36:04.0883556Z  * [new branch]              gh/rtimpe/15/head           -> origin/gh/rtimpe/15/head
2025-09-07T07:36:04.0884757Z  * [new branch]              gh/rtimpe/15/orig           -> origin/gh/rtimpe/15/orig
2025-09-07T07:36:04.0886386Z  * [new branch]              gh/rtimpe/2/base            -> origin/gh/rtimpe/2/base
2025-09-07T07:36:04.0887535Z  * [new branch]              gh/rtimpe/2/head            -> origin/gh/rtimpe/2/head
2025-09-07T07:36:04.0889134Z  * [new branch]              gh/rtimpe/3/base            -> origin/gh/rtimpe/3/base
2025-09-07T07:36:04.0890289Z  * [new branch]              gh/rtimpe/3/head            -> origin/gh/rtimpe/3/head
2025-09-07T07:36:04.0891971Z  * [new branch]              gh/rtimpe/4/base            -> origin/gh/rtimpe/4/base
2025-09-07T07:36:04.0893139Z  * [new branch]              gh/rtimpe/4/head            -> origin/gh/rtimpe/4/head
2025-09-07T07:36:04.0894929Z  * [new branch]              gh/rtimpe/9/base            -> origin/gh/rtimpe/9/base
2025-09-07T07:36:04.0896130Z  * [new branch]              gh/rtimpe/9/head            -> origin/gh/rtimpe/9/head
2025-09-07T07:36:04.0897394Z  * [new branch]              gh/rtimpe/9/orig            -> origin/gh/rtimpe/9/orig
2025-09-07T07:36:04.0899552Z  * [new branch]              gh/ruisizhang123/1/base     -> origin/gh/ruisizhang123/1/base
2025-09-07T07:36:04.0900750Z  * [new branch]              gh/ruisizhang123/1/head     -> origin/gh/ruisizhang123/1/head
2025-09-07T07:36:04.0901935Z  * [new branch]              gh/ruisizhang123/1/orig     -> origin/gh/ruisizhang123/1/orig
2025-09-07T07:36:04.0903832Z  * [new branch]              gh/ruisizhang123/4/base     -> origin/gh/ruisizhang123/4/base
2025-09-07T07:36:04.0905023Z  * [new branch]              gh/ruisizhang123/4/head     -> origin/gh/ruisizhang123/4/head
2025-09-07T07:36:04.0906188Z  * [new branch]              gh/ruisizhang123/4/orig     -> origin/gh/ruisizhang123/4/orig
2025-09-07T07:36:04.0908414Z  * [new branch]              gh/ruisizhang123/5/base     -> origin/gh/ruisizhang123/5/base
2025-09-07T07:36:04.0909740Z  * [new branch]              gh/ruisizhang123/5/head     -> origin/gh/ruisizhang123/5/head
2025-09-07T07:36:04.0910901Z  * [new branch]              gh/ruisizhang123/5/orig     -> origin/gh/ruisizhang123/5/orig
2025-09-07T07:36:04.0912738Z  * [new branch]              gh/ruisizhang123/6/base     -> origin/gh/ruisizhang123/6/base
2025-09-07T07:36:04.0913723Z  * [new branch]              gh/ruisizhang123/6/head     -> origin/gh/ruisizhang123/6/head
2025-09-07T07:36:04.0914976Z  * [new branch]              gh/ruisizhang123/6/orig     -> origin/gh/ruisizhang123/6/orig
2025-09-07T07:36:04.0916958Z  * [new branch]              gh/ruisizhang123/7/base     -> origin/gh/ruisizhang123/7/base
2025-09-07T07:36:04.0930155Z  * [new branch]              gh/ruisizhang123/7/head     -> origin/gh/ruisizhang123/7/head
2025-09-07T07:36:04.0930555Z  * [new branch]              gh/ruisizhang123/7/orig     -> origin/gh/ruisizhang123/7/orig
2025-09-07T07:36:04.0931064Z  * [new branch]              gh/ruisizhang123/8/base     -> origin/gh/ruisizhang123/8/base
2025-09-07T07:36:04.0931406Z  * [new branch]              gh/ruisizhang123/8/head     -> origin/gh/ruisizhang123/8/head
2025-09-07T07:36:04.0931742Z  * [new branch]              gh/ruisizhang123/8/orig     -> origin/gh/ruisizhang123/8/orig
2025-09-07T07:36:04.0932073Z  * [new branch]              gh/ruisizhang123/9/base     -> origin/gh/ruisizhang123/9/base
2025-09-07T07:36:04.0932397Z  * [new branch]              gh/ruisizhang123/9/head     -> origin/gh/ruisizhang123/9/head
2025-09-07T07:36:04.0932725Z  * [new branch]              gh/ruisizhang123/9/orig     -> origin/gh/ruisizhang123/9/orig
2025-09-07T07:36:04.0933050Z  * [new branch]              gh/sarckk/2/base            -> origin/gh/sarckk/2/base
2025-09-07T07:36:04.0933362Z  * [new branch]              gh/sarckk/2/head            -> origin/gh/sarckk/2/head
2025-09-07T07:36:04.0933662Z  * [new branch]              gh/sarckk/2/orig            -> origin/gh/sarckk/2/orig
2025-09-07T07:36:04.0934449Z  * [new branch]              gh/seemethere/35/base       -> origin/gh/seemethere/35/base
2025-09-07T07:36:04.0935690Z  * [new branch]              gh/seemethere/35/head       -> origin/gh/seemethere/35/head
2025-09-07T07:36:04.0936924Z  * [new branch]              gh/seemethere/35/orig       -> origin/gh/seemethere/35/orig
2025-09-07T07:36:04.0938704Z  * [new branch]              gh/seemethere/37/base       -> origin/gh/seemethere/37/base
2025-09-07T07:36:04.0939891Z  * [new branch]              gh/seemethere/37/head       -> origin/gh/seemethere/37/head
2025-09-07T07:36:04.0941054Z  * [new branch]              gh/seemethere/37/orig       -> origin/gh/seemethere/37/orig
2025-09-07T07:36:04.0942718Z  * [new branch]              gh/seemethere/43/base       -> origin/gh/seemethere/43/base
2025-09-07T07:36:04.0943996Z  * [new branch]              gh/seemethere/43/head       -> origin/gh/seemethere/43/head
2025-09-07T07:36:04.0945186Z  * [new branch]              gh/seemethere/43/orig       -> origin/gh/seemethere/43/orig
2025-09-07T07:36:04.0946906Z  * [new branch]              gh/seemethere/44/base       -> origin/gh/seemethere/44/base
2025-09-07T07:36:04.0948020Z  * [new branch]              gh/seemethere/44/head       -> origin/gh/seemethere/44/head
2025-09-07T07:36:04.0949222Z  * [new branch]              gh/seemethere/44/orig       -> origin/gh/seemethere/44/orig
2025-09-07T07:36:04.0950914Z  * [new branch]              gh/seemethere/48/base       -> origin/gh/seemethere/48/base
2025-09-07T07:36:04.0952298Z  * [new branch]              gh/seemethere/48/head       -> origin/gh/seemethere/48/head
2025-09-07T07:36:04.0953465Z  * [new branch]              gh/seemethere/48/orig       -> origin/gh/seemethere/48/orig
2025-09-07T07:36:04.0955130Z  * [new branch]              gh/seemethere/49/base       -> origin/gh/seemethere/49/base
2025-09-07T07:36:04.0956311Z  * [new branch]              gh/seemethere/49/head       -> origin/gh/seemethere/49/head
2025-09-07T07:36:04.0957582Z  * [new branch]              gh/seemethere/49/orig       -> origin/gh/seemethere/49/orig
2025-09-07T07:36:04.0959276Z  * [new branch]              gh/seemethere/52/base       -> origin/gh/seemethere/52/base
2025-09-07T07:36:04.0960581Z  * [new branch]              gh/seemethere/52/head       -> origin/gh/seemethere/52/head
2025-09-07T07:36:04.0961762Z  * [new branch]              gh/seemethere/52/orig       -> origin/gh/seemethere/52/orig
2025-09-07T07:36:04.0963456Z  * [new branch]              gh/seemethere/53/base       -> origin/gh/seemethere/53/base
2025-09-07T07:36:04.0964632Z  * [new branch]              gh/seemethere/53/head       -> origin/gh/seemethere/53/head
2025-09-07T07:36:04.0965814Z  * [new branch]              gh/seemethere/53/orig       -> origin/gh/seemethere/53/orig
2025-09-07T07:36:04.0967491Z  * [new branch]              gh/seemethere/54/base       -> origin/gh/seemethere/54/base
2025-09-07T07:36:04.0968694Z  * [new branch]              gh/seemethere/54/head       -> origin/gh/seemethere/54/head
2025-09-07T07:36:04.0969896Z  * [new branch]              gh/seemethere/54/orig       -> origin/gh/seemethere/54/orig
2025-09-07T07:36:04.0971506Z  * [new branch]              gh/seemethere/55/base       -> origin/gh/seemethere/55/base
2025-09-07T07:36:04.0972605Z  * [new branch]              gh/seemethere/55/head       -> origin/gh/seemethere/55/head
2025-09-07T07:36:04.0973834Z  * [new branch]              gh/seemethere/55/orig       -> origin/gh/seemethere/55/orig
2025-09-07T07:36:04.0975511Z  * [new branch]              gh/seemethere/56/base       -> origin/gh/seemethere/56/base
2025-09-07T07:36:04.0976787Z  * [new branch]              gh/seemethere/56/head       -> origin/gh/seemethere/56/head
2025-09-07T07:36:04.0978070Z  * [new branch]              gh/seemethere/56/orig       -> origin/gh/seemethere/56/orig
2025-09-07T07:36:04.0979782Z  * [new branch]              gh/seemethere/57/base       -> origin/gh/seemethere/57/base
2025-09-07T07:36:04.0980939Z  * [new branch]              gh/seemethere/57/head       -> origin/gh/seemethere/57/head
2025-09-07T07:36:04.0982093Z  * [new branch]              gh/seemethere/57/orig       -> origin/gh/seemethere/57/orig
2025-09-07T07:36:04.0983753Z  * [new branch]              gh/seemethere/58/base       -> origin/gh/seemethere/58/base
2025-09-07T07:36:04.0984948Z  * [new branch]              gh/seemethere/58/head       -> origin/gh/seemethere/58/head
2025-09-07T07:36:04.0986288Z  * [new branch]              gh/seemethere/58/orig       -> origin/gh/seemethere/58/orig
2025-09-07T07:36:04.0987965Z  * [new branch]              gh/seemethere/59/base       -> origin/gh/seemethere/59/base
2025-09-07T07:36:04.0989143Z  * [new branch]              gh/seemethere/59/head       -> origin/gh/seemethere/59/head
2025-09-07T07:36:04.0990333Z  * [new branch]              gh/seemethere/59/orig       -> origin/gh/seemethere/59/orig
2025-09-07T07:36:04.0992015Z  * [new branch]              gh/seemethere/60/base       -> origin/gh/seemethere/60/base
2025-09-07T07:36:04.0993243Z  * [new branch]              gh/seemethere/60/head       -> origin/gh/seemethere/60/head
2025-09-07T07:36:04.0994512Z  * [new branch]              gh/seemethere/60/orig       -> origin/gh/seemethere/60/orig
2025-09-07T07:36:04.0996120Z  * [new branch]              gh/seemethere/61/base       -> origin/gh/seemethere/61/base
2025-09-07T07:36:04.0997327Z  * [new branch]              gh/seemethere/61/head       -> origin/gh/seemethere/61/head
2025-09-07T07:36:04.0998612Z  * [new branch]              gh/seemethere/61/orig       -> origin/gh/seemethere/61/orig
2025-09-07T07:36:04.1000268Z  * [new branch]              gh/seemethere/62/base       -> origin/gh/seemethere/62/base
2025-09-07T07:36:04.1001462Z  * [new branch]              gh/seemethere/62/head       -> origin/gh/seemethere/62/head
2025-09-07T07:36:04.1002599Z  * [new branch]              gh/seemethere/62/orig       -> origin/gh/seemethere/62/orig
2025-09-07T07:36:04.1004501Z  * [new branch]              gh/seemethere/63/base       -> origin/gh/seemethere/63/base
2025-09-07T07:36:04.1005666Z  * [new branch]              gh/seemethere/63/head       -> origin/gh/seemethere/63/head
2025-09-07T07:36:04.1006880Z  * [new branch]              gh/seemethere/63/orig       -> origin/gh/seemethere/63/orig
2025-09-07T07:36:04.1009322Z  * [new branch]              gh/shunting314/145/base     -> origin/gh/shunting314/145/base
2025-09-07T07:36:04.1010571Z  * [new branch]              gh/shunting314/145/head     -> origin/gh/shunting314/145/head
2025-09-07T07:36:04.1011824Z  * [new branch]              gh/shunting314/145/orig     -> origin/gh/shunting314/145/orig
2025-09-07T07:36:04.1013745Z  * [new branch]              gh/shunting314/176/base     -> origin/gh/shunting314/176/base
2025-09-07T07:36:04.1015024Z  * [new branch]              gh/shunting314/176/head     -> origin/gh/shunting314/176/head
2025-09-07T07:36:04.1016305Z  * [new branch]              gh/shunting314/176/orig     -> origin/gh/shunting314/176/orig
2025-09-07T07:36:04.1018143Z  * [new branch]              gh/shunting314/211/base     -> origin/gh/shunting314/211/base
2025-09-07T07:36:04.1019361Z  * [new branch]              gh/shunting314/211/head     -> origin/gh/shunting314/211/head
2025-09-07T07:36:04.1020559Z  * [new branch]              gh/shunting314/211/orig     -> origin/gh/shunting314/211/orig
2025-09-07T07:36:04.1022389Z  * [new branch]              gh/shunting314/212/base     -> origin/gh/shunting314/212/base
2025-09-07T07:36:04.1023594Z  * [new branch]              gh/shunting314/212/head     -> origin/gh/shunting314/212/head
2025-09-07T07:36:04.1024797Z  * [new branch]              gh/shunting314/212/orig     -> origin/gh/shunting314/212/orig
2025-09-07T07:36:04.1026967Z  * [new branch]              gh/shunting314/213/base     -> origin/gh/shunting314/213/base
2025-09-07T07:36:04.1028191Z  * [new branch]              gh/shunting314/213/head     -> origin/gh/shunting314/213/head
2025-09-07T07:36:04.1029422Z  * [new branch]              gh/shunting314/213/orig     -> origin/gh/shunting314/213/orig
2025-09-07T07:36:04.1031125Z  * [new branch]              gh/shunting314/214/base     -> origin/gh/shunting314/214/base
2025-09-07T07:36:04.1032346Z  * [new branch]              gh/shunting314/214/head     -> origin/gh/shunting314/214/head
2025-09-07T07:36:04.1033521Z  * [new branch]              gh/shunting314/214/orig     -> origin/gh/shunting314/214/orig
2025-09-07T07:36:04.1035363Z  * [new branch]              gh/shunting314/215/base     -> origin/gh/shunting314/215/base
2025-09-07T07:36:04.1036628Z  * [new branch]              gh/shunting314/215/head     -> origin/gh/shunting314/215/head
2025-09-07T07:36:04.1037821Z  * [new branch]              gh/shunting314/215/orig     -> origin/gh/shunting314/215/orig
2025-09-07T07:36:04.1039481Z  * [new branch]              gh/shunting314/216/base     -> origin/gh/shunting314/216/base
2025-09-07T07:36:04.1040644Z  * [new branch]              gh/shunting314/216/head     -> origin/gh/shunting314/216/head
2025-09-07T07:36:04.1041799Z  * [new branch]              gh/shunting314/216/orig     -> origin/gh/shunting314/216/orig
2025-09-07T07:36:04.1044182Z  * [new branch]              gh/shunting314/217/base     -> origin/gh/shunting314/217/base
2025-09-07T07:36:04.1045294Z  * [new branch]              gh/shunting314/217/head     -> origin/gh/shunting314/217/head
2025-09-07T07:36:04.1046484Z  * [new branch]              gh/shunting314/217/orig     -> origin/gh/shunting314/217/orig
2025-09-07T07:36:04.1048260Z  * [new branch]              gh/shunting314/218/base     -> origin/gh/shunting314/218/base
2025-09-07T07:36:04.1049432Z  * [new branch]              gh/shunting314/218/head     -> origin/gh/shunting314/218/head
2025-09-07T07:36:04.1050608Z  * [new branch]              gh/shunting314/218/orig     -> origin/gh/shunting314/218/orig
2025-09-07T07:36:04.1052206Z  * [new branch]              gh/shunting314/219/base     -> origin/gh/shunting314/219/base
2025-09-07T07:36:04.1053378Z  * [new branch]              gh/shunting314/219/head     -> origin/gh/shunting314/219/head
2025-09-07T07:36:04.1054548Z  * [new branch]              gh/shunting314/219/orig     -> origin/gh/shunting314/219/orig
2025-09-07T07:36:04.1056359Z  * [new branch]              gh/shunting314/220/base     -> origin/gh/shunting314/220/base
2025-09-07T07:36:04.1057945Z  * [new branch]              gh/shunting314/220/head     -> origin/gh/shunting314/220/head
2025-09-07T07:36:04.1059172Z  * [new branch]              gh/shunting314/220/orig     -> origin/gh/shunting314/220/orig
2025-09-07T07:36:04.1060875Z  * [new branch]              gh/shunting314/221/base     -> origin/gh/shunting314/221/base
2025-09-07T07:36:04.1062444Z  * [new branch]              gh/shunting314/221/head     -> origin/gh/shunting314/221/head
2025-09-07T07:36:04.1063680Z  * [new branch]              gh/shunting314/221/orig     -> origin/gh/shunting314/221/orig
2025-09-07T07:36:04.1065703Z  * [new branch]              gh/shunting314/222/base     -> origin/gh/shunting314/222/base
2025-09-07T07:36:04.1066870Z  * [new branch]              gh/shunting314/222/head     -> origin/gh/shunting314/222/head
2025-09-07T07:36:04.1068061Z  * [new branch]              gh/shunting314/222/orig     -> origin/gh/shunting314/222/orig
2025-09-07T07:36:04.1069976Z  * [new branch]              gh/shunting314/223/base     -> origin/gh/shunting314/223/base
2025-09-07T07:36:04.1071242Z  * [new branch]              gh/shunting314/223/head     -> origin/gh/shunting314/223/head
2025-09-07T07:36:04.1072431Z  * [new branch]              gh/shunting314/223/orig     -> origin/gh/shunting314/223/orig
2025-09-07T07:36:04.1074607Z  * [new branch]              gh/silverguo/1/base         -> origin/gh/silverguo/1/base
2025-09-07T07:36:04.1075896Z  * [new branch]              gh/silverguo/1/head         -> origin/gh/silverguo/1/head
2025-09-07T07:36:04.1077902Z  * [new branch]              gh/silverguo/2/base         -> origin/gh/silverguo/2/base
2025-09-07T07:36:04.1079034Z  * [new branch]              gh/silverguo/2/head         -> origin/gh/silverguo/2/head
2025-09-07T07:36:04.1080622Z  * [new branch]              gh/silverguo/3/base         -> origin/gh/silverguo/3/base
2025-09-07T07:36:04.1081808Z  * [new branch]              gh/silverguo/3/head         -> origin/gh/silverguo/3/head
2025-09-07T07:36:04.1083379Z  * [new branch]              gh/silverguo/4/base         -> origin/gh/silverguo/4/base
2025-09-07T07:36:04.1084549Z  * [new branch]              gh/silverguo/4/head         -> origin/gh/silverguo/4/head
2025-09-07T07:36:04.1086735Z  * [new branch]              gh/sinhaanhsul/1/base       -> origin/gh/sinhaanhsul/1/base
2025-09-07T07:36:04.1087921Z  * [new branch]              gh/sinhaanhsul/1/head       -> origin/gh/sinhaanhsul/1/head
2025-09-07T07:36:04.1090049Z  * [new branch]              gh/skarjala/17/base         -> origin/gh/skarjala/17/base
2025-09-07T07:36:04.1091224Z  * [new branch]              gh/skarjala/17/head         -> origin/gh/skarjala/17/head
2025-09-07T07:36:04.1092554Z  * [new branch]              gh/skarjala/17/orig         -> origin/gh/skarjala/17/orig
2025-09-07T07:36:04.1094764Z  * [new branch]              gh/skarjala/18/base         -> origin/gh/skarjala/18/base
2025-09-07T07:36:04.1096151Z  * [new branch]              gh/skarjala/18/head         -> origin/gh/skarjala/18/head
2025-09-07T07:36:04.1097256Z  * [new branch]              gh/skarjala/18/orig         -> origin/gh/skarjala/18/orig
2025-09-07T07:36:04.1099059Z  * [new branch]              gh/skarjala/19/base         -> origin/gh/skarjala/19/base
2025-09-07T07:36:04.1100256Z  * [new branch]              gh/skarjala/19/head         -> origin/gh/skarjala/19/head
2025-09-07T07:36:04.1101469Z  * [new branch]              gh/skarjala/19/orig         -> origin/gh/skarjala/19/orig
2025-09-07T07:36:04.1103809Z  * [new branch]              gh/slayton58/1/base         -> origin/gh/slayton58/1/base
2025-09-07T07:36:04.1105021Z  * [new branch]              gh/slayton58/1/head         -> origin/gh/slayton58/1/head
2025-09-07T07:36:04.1106214Z  * [new branch]              gh/slayton58/1/orig         -> origin/gh/slayton58/1/orig
2025-09-07T07:36:04.1107896Z  * [new branch]              gh/slayton58/2/base         -> origin/gh/slayton58/2/base
2025-09-07T07:36:04.1109187Z  * [new branch]              gh/slayton58/2/head         -> origin/gh/slayton58/2/head
2025-09-07T07:36:04.1110466Z  * [new branch]              gh/slayton58/2/orig         -> origin/gh/slayton58/2/orig
2025-09-07T07:36:04.1112155Z  * [new branch]              gh/slayton58/3/base         -> origin/gh/slayton58/3/base
2025-09-07T07:36:04.1113280Z  * [new branch]              gh/slayton58/3/head         -> origin/gh/slayton58/3/head
2025-09-07T07:36:04.1114476Z  * [new branch]              gh/slayton58/3/orig         -> origin/gh/slayton58/3/orig
2025-09-07T07:36:04.1116168Z  * [new branch]              gh/slayton58/4/base         -> origin/gh/slayton58/4/base
2025-09-07T07:36:04.1117306Z  * [new branch]              gh/slayton58/4/head         -> origin/gh/slayton58/4/head
2025-09-07T07:36:04.1118512Z  * [new branch]              gh/slayton58/4/orig         -> origin/gh/slayton58/4/orig
2025-09-07T07:36:04.1120219Z  * [new branch]              gh/slayton58/5/base         -> origin/gh/slayton58/5/base
2025-09-07T07:36:04.1121435Z  * [new branch]              gh/slayton58/5/head         -> origin/gh/slayton58/5/head
2025-09-07T07:36:04.1122642Z  * [new branch]              gh/slayton58/5/orig         -> origin/gh/slayton58/5/orig
2025-09-07T07:36:04.1125047Z  * [new branch]              gh/soulitzer/269/base       -> origin/gh/soulitzer/269/base
2025-09-07T07:36:04.1126201Z  * [new branch]              gh/soulitzer/269/head       -> origin/gh/soulitzer/269/head
2025-09-07T07:36:04.1127426Z  * [new branch]              gh/soulitzer/269/orig       -> origin/gh/soulitzer/269/orig
2025-09-07T07:36:04.1129216Z  * [new branch]              gh/soulitzer/276/base       -> origin/gh/soulitzer/276/base
2025-09-07T07:36:04.1130463Z  * [new branch]              gh/soulitzer/276/head       -> origin/gh/soulitzer/276/head
2025-09-07T07:36:04.1131814Z  * [new branch]              gh/soulitzer/276/orig       -> origin/gh/soulitzer/276/orig
2025-09-07T07:36:04.1133733Z  * [new branch]              gh/soulitzer/287/base       -> origin/gh/soulitzer/287/base
2025-09-07T07:36:04.1134879Z  * [new branch]              gh/soulitzer/287/head       -> origin/gh/soulitzer/287/head
2025-09-07T07:36:04.1136531Z  * [new branch]              gh/soulitzer/287/orig       -> origin/gh/soulitzer/287/orig
2025-09-07T07:36:04.1138401Z  * [new branch]              gh/soulitzer/296/base       -> origin/gh/soulitzer/296/base
2025-09-07T07:36:04.1139605Z  * [new branch]              gh/soulitzer/296/head       -> origin/gh/soulitzer/296/head
2025-09-07T07:36:04.1140846Z  * [new branch]              gh/soulitzer/296/orig       -> origin/gh/soulitzer/296/orig
2025-09-07T07:36:04.1142666Z  * [new branch]              gh/soulitzer/299/base       -> origin/gh/soulitzer/299/base
2025-09-07T07:36:04.1143933Z  * [new branch]              gh/soulitzer/299/head       -> origin/gh/soulitzer/299/head
2025-09-07T07:36:04.1145110Z  * [new branch]              gh/soulitzer/299/orig       -> origin/gh/soulitzer/299/orig
2025-09-07T07:36:04.1146905Z  * [new branch]              gh/soulitzer/300/base       -> origin/gh/soulitzer/300/base
2025-09-07T07:36:04.1148065Z  * [new branch]              gh/soulitzer/300/head       -> origin/gh/soulitzer/300/head
2025-09-07T07:36:04.1149248Z  * [new branch]              gh/soulitzer/300/orig       -> origin/gh/soulitzer/300/orig
2025-09-07T07:36:04.1151053Z  * [new branch]              gh/soulitzer/301/base       -> origin/gh/soulitzer/301/base
2025-09-07T07:36:04.1152292Z  * [new branch]              gh/soulitzer/301/head       -> origin/gh/soulitzer/301/head
2025-09-07T07:36:04.1153479Z  * [new branch]              gh/soulitzer/301/orig       -> origin/gh/soulitzer/301/orig
2025-09-07T07:36:04.1155177Z  * [new branch]              gh/soulitzer/313/base       -> origin/gh/soulitzer/313/base
2025-09-07T07:36:04.1156359Z  * [new branch]              gh/soulitzer/313/head       -> origin/gh/soulitzer/313/head
2025-09-07T07:36:04.1157538Z  * [new branch]              gh/soulitzer/313/orig       -> origin/gh/soulitzer/313/orig
2025-09-07T07:36:04.1159861Z  * [new branch]              gh/soulitzer/319/base       -> origin/gh/soulitzer/319/base
2025-09-07T07:36:04.1161079Z  * [new branch]              gh/soulitzer/319/head       -> origin/gh/soulitzer/319/head
2025-09-07T07:36:04.1162272Z  * [new branch]              gh/soulitzer/319/orig       -> origin/gh/soulitzer/319/orig
2025-09-07T07:36:04.1164097Z  * [new branch]              gh/soulitzer/320/base       -> origin/gh/soulitzer/320/base
2025-09-07T07:36:04.1165222Z  * [new branch]              gh/soulitzer/320/head       -> origin/gh/soulitzer/320/head
2025-09-07T07:36:04.1166411Z  * [new branch]              gh/soulitzer/320/orig       -> origin/gh/soulitzer/320/orig
2025-09-07T07:36:04.1168171Z  * [new branch]              gh/soulitzer/336/base       -> origin/gh/soulitzer/336/base
2025-09-07T07:36:04.1169462Z  * [new branch]              gh/soulitzer/336/head       -> origin/gh/soulitzer/336/head
2025-09-07T07:36:04.1170643Z  * [new branch]              gh/soulitzer/336/orig       -> origin/gh/soulitzer/336/orig
2025-09-07T07:36:04.1172358Z  * [new branch]              gh/soulitzer/347/base       -> origin/gh/soulitzer/347/base
2025-09-07T07:36:04.1173491Z  * [new branch]              gh/soulitzer/347/head       -> origin/gh/soulitzer/347/head
2025-09-07T07:36:04.1174661Z  * [new branch]              gh/soulitzer/347/orig       -> origin/gh/soulitzer/347/orig
2025-09-07T07:36:04.1176605Z  * [new branch]              gh/soulitzer/349/base       -> origin/gh/soulitzer/349/base
2025-09-07T07:36:04.1177924Z  * [new branch]              gh/soulitzer/349/head       -> origin/gh/soulitzer/349/head
2025-09-07T07:36:04.1179200Z  * [new branch]              gh/soulitzer/349/orig       -> origin/gh/soulitzer/349/orig
2025-09-07T07:36:04.1180843Z  * [new branch]              gh/soulitzer/350/base       -> origin/gh/soulitzer/350/base
2025-09-07T07:36:04.1181956Z  * [new branch]              gh/soulitzer/350/head       -> origin/gh/soulitzer/350/head
2025-09-07T07:36:04.1183154Z  * [new branch]              gh/soulitzer/350/orig       -> origin/gh/soulitzer/350/orig
2025-09-07T07:36:04.1185344Z  * [new branch]              gh/soulitzer/351/base       -> origin/gh/soulitzer/351/base
2025-09-07T07:36:04.1186530Z  * [new branch]              gh/soulitzer/351/head       -> origin/gh/soulitzer/351/head
2025-09-07T07:36:04.1187709Z  * [new branch]              gh/soulitzer/351/orig       -> origin/gh/soulitzer/351/orig
2025-09-07T07:36:04.1189386Z  * [new branch]              gh/soulitzer/353/base       -> origin/gh/soulitzer/353/base
2025-09-07T07:36:04.1190680Z  * [new branch]              gh/soulitzer/353/head       -> origin/gh/soulitzer/353/head
2025-09-07T07:36:04.1191860Z  * [new branch]              gh/soulitzer/353/orig       -> origin/gh/soulitzer/353/orig
2025-09-07T07:36:04.1194183Z  * [new branch]              gh/soulitzer/358/base       -> origin/gh/soulitzer/358/base
2025-09-07T07:36:04.1195462Z  * [new branch]              gh/soulitzer/358/head       -> origin/gh/soulitzer/358/head
2025-09-07T07:36:04.1196732Z  * [new branch]              gh/soulitzer/358/orig       -> origin/gh/soulitzer/358/orig
2025-09-07T07:36:04.1198800Z  * [new branch]              gh/soulitzer/359/base       -> origin/gh/soulitzer/359/base
2025-09-07T07:36:04.1200008Z  * [new branch]              gh/soulitzer/359/head       -> origin/gh/soulitzer/359/head
2025-09-07T07:36:04.1201215Z  * [new branch]              gh/soulitzer/359/orig       -> origin/gh/soulitzer/359/orig
2025-09-07T07:36:04.1202957Z  * [new branch]              gh/soulitzer/362/base       -> origin/gh/soulitzer/362/base
2025-09-07T07:36:04.1204394Z  * [new branch]              gh/soulitzer/362/head       -> origin/gh/soulitzer/362/head
2025-09-07T07:36:04.1205828Z  * [new branch]              gh/soulitzer/362/orig       -> origin/gh/soulitzer/362/orig
2025-09-07T07:36:04.1207591Z  * [new branch]              gh/soulitzer/372/base       -> origin/gh/soulitzer/372/base
2025-09-07T07:36:04.1208759Z  * [new branch]              gh/soulitzer/372/head       -> origin/gh/soulitzer/372/head
2025-09-07T07:36:04.1209926Z  * [new branch]              gh/soulitzer/372/orig       -> origin/gh/soulitzer/372/orig
2025-09-07T07:36:04.1211877Z  * [new branch]              gh/soulitzer/373/base       -> origin/gh/soulitzer/373/base
2025-09-07T07:36:04.1213038Z  * [new branch]              gh/soulitzer/373/head       -> origin/gh/soulitzer/373/head
2025-09-07T07:36:04.1214238Z  * [new branch]              gh/soulitzer/373/orig       -> origin/gh/soulitzer/373/orig
2025-09-07T07:36:04.1216047Z  * [new branch]              gh/soulitzer/374/base       -> origin/gh/soulitzer/374/base
2025-09-07T07:36:04.1217320Z  * [new branch]              gh/soulitzer/374/head       -> origin/gh/soulitzer/374/head
2025-09-07T07:36:04.1218561Z  * [new branch]              gh/soulitzer/374/orig       -> origin/gh/soulitzer/374/orig
2025-09-07T07:36:04.1220323Z  * [new branch]              gh/soulitzer/375/base       -> origin/gh/soulitzer/375/base
2025-09-07T07:36:04.1221456Z  * [new branch]              gh/soulitzer/375/head       -> origin/gh/soulitzer/375/head
2025-09-07T07:36:04.1222651Z  * [new branch]              gh/soulitzer/375/orig       -> origin/gh/soulitzer/375/orig
2025-09-07T07:36:04.1224374Z  * [new branch]              gh/soulitzer/376/base       -> origin/gh/soulitzer/376/base
2025-09-07T07:36:04.1225590Z  * [new branch]              gh/soulitzer/376/head       -> origin/gh/soulitzer/376/head
2025-09-07T07:36:04.1226769Z  * [new branch]              gh/soulitzer/376/orig       -> origin/gh/soulitzer/376/orig
2025-09-07T07:36:04.1228682Z  * [new branch]              gh/soulitzer/377/base       -> origin/gh/soulitzer/377/base
2025-09-07T07:36:04.1230249Z  * [new branch]              gh/soulitzer/377/head       -> origin/gh/soulitzer/377/head
2025-09-07T07:36:04.1231559Z  * [new branch]              gh/soulitzer/377/orig       -> origin/gh/soulitzer/377/orig
2025-09-07T07:36:04.1233371Z  * [new branch]              gh/soulitzer/378/base       -> origin/gh/soulitzer/378/base
2025-09-07T07:36:04.1234564Z  * [new branch]              gh/soulitzer/378/head       -> origin/gh/soulitzer/378/head
2025-09-07T07:36:04.1235824Z  * [new branch]              gh/soulitzer/378/orig       -> origin/gh/soulitzer/378/orig
2025-09-07T07:36:04.1237584Z  * [new branch]              gh/soulitzer/379/base       -> origin/gh/soulitzer/379/base
2025-09-07T07:36:04.1238847Z  * [new branch]              gh/soulitzer/379/head       -> origin/gh/soulitzer/379/head
2025-09-07T07:36:04.1240029Z  * [new branch]              gh/soulitzer/379/orig       -> origin/gh/soulitzer/379/orig
2025-09-07T07:36:04.1242857Z  * [new branch]              gh/swolchok/728/next        -> origin/gh/swolchok/728/next
2025-09-07T07:36:04.1244762Z  * [new branch]              gh/swolchok/767/base        -> origin/gh/swolchok/767/base
2025-09-07T07:36:04.1246276Z  * [new branch]              gh/swolchok/767/head        -> origin/gh/swolchok/767/head
2025-09-07T07:36:04.1247736Z  * [new branch]              gh/swolchok/767/orig        -> origin/gh/swolchok/767/orig
2025-09-07T07:36:04.1249406Z  * [new branch]              gh/swolchok/768/base        -> origin/gh/swolchok/768/base
2025-09-07T07:36:04.1250709Z  * [new branch]              gh/swolchok/768/head        -> origin/gh/swolchok/768/head
2025-09-07T07:36:04.1251921Z  * [new branch]              gh/swolchok/768/orig        -> origin/gh/swolchok/768/orig
2025-09-07T07:36:04.1253780Z  * [new branch]              gh/swolchok/769/base        -> origin/gh/swolchok/769/base
2025-09-07T07:36:04.1255118Z  * [new branch]              gh/swolchok/769/head        -> origin/gh/swolchok/769/head
2025-09-07T07:36:04.1256520Z  * [new branch]              gh/swolchok/769/orig        -> origin/gh/swolchok/769/orig
2025-09-07T07:36:04.1258280Z  * [new branch]              gh/swolchok/771/base        -> origin/gh/swolchok/771/base
2025-09-07T07:36:04.1259659Z  * [new branch]              gh/swolchok/771/head        -> origin/gh/swolchok/771/head
2025-09-07T07:36:04.1260951Z  * [new branch]              gh/swolchok/771/orig        -> origin/gh/swolchok/771/orig
2025-09-07T07:36:04.1262878Z  * [new branch]              gh/swolchok/772/base        -> origin/gh/swolchok/772/base
2025-09-07T07:36:04.1264164Z  * [new branch]              gh/swolchok/772/head        -> origin/gh/swolchok/772/head
2025-09-07T07:36:04.1265405Z  * [new branch]              gh/swolchok/772/orig        -> origin/gh/swolchok/772/orig
2025-09-07T07:36:04.1267257Z  * [new branch]              gh/swolchok/773/base        -> origin/gh/swolchok/773/base
2025-09-07T07:36:04.1268458Z  * [new branch]              gh/swolchok/773/head        -> origin/gh/swolchok/773/head
2025-09-07T07:36:04.1269687Z  * [new branch]              gh/swolchok/773/orig        -> origin/gh/swolchok/773/orig
2025-09-07T07:36:04.1271440Z  * [new branch]              gh/swolchok/786/base        -> origin/gh/swolchok/786/base
2025-09-07T07:36:04.1272602Z  * [new branch]              gh/swolchok/786/head        -> origin/gh/swolchok/786/head
2025-09-07T07:36:04.1273860Z  * [new branch]              gh/swolchok/786/orig        -> origin/gh/swolchok/786/orig
2025-09-07T07:36:04.1275461Z  * [new branch]              gh/swolchok/787/base        -> origin/gh/swolchok/787/base
2025-09-07T07:36:04.1276632Z  * [new branch]              gh/swolchok/787/head        -> origin/gh/swolchok/787/head
2025-09-07T07:36:04.1277877Z  * [new branch]              gh/swolchok/787/orig        -> origin/gh/swolchok/787/orig
2025-09-07T07:36:04.1279744Z  * [new branch]              gh/swolchok/788/base        -> origin/gh/swolchok/788/base
2025-09-07T07:36:04.1281019Z  * [new branch]              gh/swolchok/788/head        -> origin/gh/swolchok/788/head
2025-09-07T07:36:04.1282186Z  * [new branch]              gh/swolchok/788/orig        -> origin/gh/swolchok/788/orig
2025-09-07T07:36:04.1283861Z  * [new branch]              gh/swolchok/789/base        -> origin/gh/swolchok/789/base
2025-09-07T07:36:04.1285021Z  * [new branch]              gh/swolchok/789/head        -> origin/gh/swolchok/789/head
2025-09-07T07:36:04.1286305Z  * [new branch]              gh/swolchok/789/orig        -> origin/gh/swolchok/789/orig
2025-09-07T07:36:04.1287989Z  * [new branch]              gh/swolchok/790/base        -> origin/gh/swolchok/790/base
2025-09-07T07:36:04.1289199Z  * [new branch]              gh/swolchok/790/head        -> origin/gh/swolchok/790/head
2025-09-07T07:36:04.1290384Z  * [new branch]              gh/swolchok/790/orig        -> origin/gh/swolchok/790/orig
2025-09-07T07:36:04.1292172Z  * [new branch]              gh/swolchok/791/base        -> origin/gh/swolchok/791/base
2025-09-07T07:36:04.1293278Z  * [new branch]              gh/swolchok/791/head        -> origin/gh/swolchok/791/head
2025-09-07T07:36:04.1294532Z  * [new branch]              gh/swolchok/791/orig        -> origin/gh/swolchok/791/orig
2025-09-07T07:36:04.1296240Z  * [new branch]              gh/swolchok/792/base        -> origin/gh/swolchok/792/base
2025-09-07T07:36:04.1297753Z  * [new branch]              gh/swolchok/792/head        -> origin/gh/swolchok/792/head
2025-09-07T07:36:04.1299204Z  * [new branch]              gh/swolchok/792/orig        -> origin/gh/swolchok/792/orig
2025-09-07T07:36:04.1300986Z  * [new branch]              gh/swolchok/793/base        -> origin/gh/swolchok/793/base
2025-09-07T07:36:04.1302124Z  * [new branch]              gh/swolchok/793/head        -> origin/gh/swolchok/793/head
2025-09-07T07:36:04.1303558Z  * [new branch]              gh/swolchok/793/orig        -> origin/gh/swolchok/793/orig
2025-09-07T07:36:04.1305363Z  * [new branch]              gh/swolchok/794/base        -> origin/gh/swolchok/794/base
2025-09-07T07:36:04.1306543Z  * [new branch]              gh/swolchok/794/head        -> origin/gh/swolchok/794/head
2025-09-07T07:36:04.1307664Z  * [new branch]              gh/swolchok/794/orig        -> origin/gh/swolchok/794/orig
2025-09-07T07:36:04.1309856Z  * [new branch]              gh/swolchok/795/base        -> origin/gh/swolchok/795/base
2025-09-07T07:36:04.1311091Z  * [new branch]              gh/swolchok/795/head        -> origin/gh/swolchok/795/head
2025-09-07T07:36:04.1312330Z  * [new branch]              gh/swolchok/795/orig        -> origin/gh/swolchok/795/orig
2025-09-07T07:36:04.1314098Z  * [new branch]              gh/swolchok/796/base        -> origin/gh/swolchok/796/base
2025-09-07T07:36:04.1315620Z  * [new branch]              gh/swolchok/796/head        -> origin/gh/swolchok/796/head
2025-09-07T07:36:04.1316815Z  * [new branch]              gh/swolchok/796/orig        -> origin/gh/swolchok/796/orig
2025-09-07T07:36:04.1318849Z  * [new branch]              gh/swolchok/797/base        -> origin/gh/swolchok/797/base
2025-09-07T07:36:04.1320591Z  * [new branch]              gh/swolchok/797/head        -> origin/gh/swolchok/797/head
2025-09-07T07:36:04.1321871Z  * [new branch]              gh/swolchok/797/orig        -> origin/gh/swolchok/797/orig
2025-09-07T07:36:04.1323688Z  * [new branch]              gh/swolchok/798/base        -> origin/gh/swolchok/798/base
2025-09-07T07:36:04.1324812Z  * [new branch]              gh/swolchok/798/head        -> origin/gh/swolchok/798/head
2025-09-07T07:36:04.1326098Z  * [new branch]              gh/swolchok/798/orig        -> origin/gh/swolchok/798/orig
2025-09-07T07:36:04.1328404Z  * [new branch]              gh/swolchok/799/base        -> origin/gh/swolchok/799/base
2025-09-07T07:36:04.1329544Z  * [new branch]              gh/swolchok/799/head        -> origin/gh/swolchok/799/head
2025-09-07T07:36:04.1330863Z  * [new branch]              gh/swolchok/799/orig        -> origin/gh/swolchok/799/orig
2025-09-07T07:36:04.1332760Z  * [new branch]              gh/swolchok/800/base        -> origin/gh/swolchok/800/base
2025-09-07T07:36:04.1333974Z  * [new branch]              gh/swolchok/800/head        -> origin/gh/swolchok/800/head
2025-09-07T07:36:04.1335260Z  * [new branch]              gh/swolchok/800/orig        -> origin/gh/swolchok/800/orig
2025-09-07T07:36:04.1337093Z  * [new branch]              gh/swolchok/801/base        -> origin/gh/swolchok/801/base
2025-09-07T07:36:04.1338341Z  * [new branch]              gh/swolchok/801/head        -> origin/gh/swolchok/801/head
2025-09-07T07:36:04.1339658Z  * [new branch]              gh/swolchok/801/orig        -> origin/gh/swolchok/801/orig
2025-09-07T07:36:04.1341429Z  * [new branch]              gh/swolchok/802/base        -> origin/gh/swolchok/802/base
2025-09-07T07:36:04.1342548Z  * [new branch]              gh/swolchok/802/head        -> origin/gh/swolchok/802/head
2025-09-07T07:36:04.1343792Z  * [new branch]              gh/swolchok/802/orig        -> origin/gh/swolchok/802/orig
2025-09-07T07:36:04.1345541Z  * [new branch]              gh/swolchok/803/base        -> origin/gh/swolchok/803/base
2025-09-07T07:36:04.1346733Z  * [new branch]              gh/swolchok/803/head        -> origin/gh/swolchok/803/head
2025-09-07T07:36:04.1348001Z  * [new branch]              gh/swolchok/803/orig        -> origin/gh/swolchok/803/orig
2025-09-07T07:36:04.1349927Z  * [new branch]              gh/swolchok/804/base        -> origin/gh/swolchok/804/base
2025-09-07T07:36:04.1351001Z  * [new branch]              gh/swolchok/804/head        -> origin/gh/swolchok/804/head
2025-09-07T07:36:04.1352403Z  * [new branch]              gh/swolchok/804/orig        -> origin/gh/swolchok/804/orig
2025-09-07T07:36:04.1354190Z  * [new branch]              gh/swolchok/805/base        -> origin/gh/swolchok/805/base
2025-09-07T07:36:04.1355398Z  * [new branch]              gh/swolchok/805/head        -> origin/gh/swolchok/805/head
2025-09-07T07:36:04.1356717Z  * [new branch]              gh/swolchok/805/orig        -> origin/gh/swolchok/805/orig
2025-09-07T07:36:04.1358386Z  * [new branch]              gh/swolchok/806/base        -> origin/gh/swolchok/806/base
2025-09-07T07:36:04.1359626Z  * [new branch]              gh/swolchok/806/head        -> origin/gh/swolchok/806/head
2025-09-07T07:36:04.1360929Z  * [new branch]              gh/swolchok/806/orig        -> origin/gh/swolchok/806/orig
2025-09-07T07:36:04.1362730Z  * [new branch]              gh/swolchok/807/base        -> origin/gh/swolchok/807/base
2025-09-07T07:36:04.1363984Z  * [new branch]              gh/swolchok/807/head        -> origin/gh/swolchok/807/head
2025-09-07T07:36:04.1365421Z  * [new branch]              gh/swolchok/807/orig        -> origin/gh/swolchok/807/orig
2025-09-07T07:36:04.1367459Z  * [new branch]              gh/swolchok/808/base        -> origin/gh/swolchok/808/base
2025-09-07T07:36:04.1368769Z  * [new branch]              gh/swolchok/808/head        -> origin/gh/swolchok/808/head
2025-09-07T07:36:04.1369960Z  * [new branch]              gh/swolchok/808/orig        -> origin/gh/swolchok/808/orig
2025-09-07T07:36:04.1371759Z  * [new branch]              gh/swolchok/809/base        -> origin/gh/swolchok/809/base
2025-09-07T07:36:04.1372947Z  * [new branch]              gh/swolchok/809/head        -> origin/gh/swolchok/809/head
2025-09-07T07:36:04.1374149Z  * [new branch]              gh/swolchok/809/orig        -> origin/gh/swolchok/809/orig
2025-09-07T07:36:04.1376063Z  * [new branch]              gh/swolchok/810/base        -> origin/gh/swolchok/810/base
2025-09-07T07:36:04.1377186Z  * [new branch]              gh/swolchok/810/head        -> origin/gh/swolchok/810/head
2025-09-07T07:36:04.1378491Z  * [new branch]              gh/swolchok/810/orig        -> origin/gh/swolchok/810/orig
2025-09-07T07:36:04.1380285Z  * [new branch]              gh/swolchok/811/base        -> origin/gh/swolchok/811/base
2025-09-07T07:36:04.1381555Z  * [new branch]              gh/swolchok/811/head        -> origin/gh/swolchok/811/head
2025-09-07T07:36:04.1382784Z  * [new branch]              gh/swolchok/811/orig        -> origin/gh/swolchok/811/orig
2025-09-07T07:36:04.1384598Z  * [new branch]              gh/swolchok/812/base        -> origin/gh/swolchok/812/base
2025-09-07T07:36:04.1385778Z  * [new branch]              gh/swolchok/812/head        -> origin/gh/swolchok/812/head
2025-09-07T07:36:04.1387414Z  * [new branch]              gh/swolchok/812/orig        -> origin/gh/swolchok/812/orig
2025-09-07T07:36:04.1389362Z  * [new branch]              gh/swolchok/813/base        -> origin/gh/swolchok/813/base
2025-09-07T07:36:04.1390684Z  * [new branch]              gh/swolchok/813/head        -> origin/gh/swolchok/813/head
2025-09-07T07:36:04.1391876Z  * [new branch]              gh/swolchok/813/orig        -> origin/gh/swolchok/813/orig
2025-09-07T07:36:04.1393756Z  * [new branch]              gh/swolchok/814/base        -> origin/gh/swolchok/814/base
2025-09-07T07:36:04.1394881Z  * [new branch]              gh/swolchok/814/head        -> origin/gh/swolchok/814/head
2025-09-07T07:36:04.1396073Z  * [new branch]              gh/swolchok/814/orig        -> origin/gh/swolchok/814/orig
2025-09-07T07:36:04.1397973Z  * [new branch]              gh/swolchok/815/base        -> origin/gh/swolchok/815/base
2025-09-07T07:36:04.1399141Z  * [new branch]              gh/swolchok/815/head        -> origin/gh/swolchok/815/head
2025-09-07T07:36:04.1400483Z  * [new branch]              gh/swolchok/815/orig        -> origin/gh/swolchok/815/orig
2025-09-07T07:36:04.1402224Z  * [new branch]              gh/swolchok/816/base        -> origin/gh/swolchok/816/base
2025-09-07T07:36:04.1403712Z  * [new branch]              gh/swolchok/816/head        -> origin/gh/swolchok/816/head
2025-09-07T07:36:04.1405414Z  * [new branch]              gh/swolchok/816/orig        -> origin/gh/swolchok/816/orig
2025-09-07T07:36:04.1407298Z  * [new branch]              gh/swolchok/817/base        -> origin/gh/swolchok/817/base
2025-09-07T07:36:04.1408514Z  * [new branch]              gh/swolchok/817/head        -> origin/gh/swolchok/817/head
2025-09-07T07:36:04.1409636Z  * [new branch]              gh/swolchok/817/orig        -> origin/gh/swolchok/817/orig
2025-09-07T07:36:04.1411423Z  * [new branch]              gh/swolchok/818/base        -> origin/gh/swolchok/818/base
2025-09-07T07:36:04.1412539Z  * [new branch]              gh/swolchok/818/head        -> origin/gh/swolchok/818/head
2025-09-07T07:36:04.1413860Z  * [new branch]              gh/swolchok/818/orig        -> origin/gh/swolchok/818/orig
2025-09-07T07:36:04.1416102Z  * [new branch]              gh/swolchok/819/base        -> origin/gh/swolchok/819/base
2025-09-07T07:36:04.1417245Z  * [new branch]              gh/swolchok/819/head        -> origin/gh/swolchok/819/head
2025-09-07T07:36:04.1418515Z  * [new branch]              gh/swolchok/819/orig        -> origin/gh/swolchok/819/orig
2025-09-07T07:36:04.1420328Z  * [new branch]              gh/swolchok/820/base        -> origin/gh/swolchok/820/base
2025-09-07T07:36:04.1421565Z  * [new branch]              gh/swolchok/820/head        -> origin/gh/swolchok/820/head
2025-09-07T07:36:04.1423146Z  * [new branch]              gh/swolchok/820/orig        -> origin/gh/swolchok/820/orig
2025-09-07T07:36:04.1424990Z  * [new branch]              gh/swolchok/821/base        -> origin/gh/swolchok/821/base
2025-09-07T07:36:04.1426137Z  * [new branch]              gh/swolchok/821/head        -> origin/gh/swolchok/821/head
2025-09-07T07:36:04.1427511Z  * [new branch]              gh/swolchok/821/orig        -> origin/gh/swolchok/821/orig
2025-09-07T07:36:04.1429409Z  * [new branch]              gh/swolchok/822/base        -> origin/gh/swolchok/822/base
2025-09-07T07:36:04.1430562Z  * [new branch]              gh/swolchok/822/head        -> origin/gh/swolchok/822/head
2025-09-07T07:36:04.1431746Z  * [new branch]              gh/swolchok/822/orig        -> origin/gh/swolchok/822/orig
2025-09-07T07:36:04.1433713Z  * [new branch]              gh/swolchok/823/base        -> origin/gh/swolchok/823/base
2025-09-07T07:36:04.1434858Z  * [new branch]              gh/swolchok/823/head        -> origin/gh/swolchok/823/head
2025-09-07T07:36:04.1436063Z  * [new branch]              gh/swolchok/823/orig        -> origin/gh/swolchok/823/orig
2025-09-07T07:36:04.1437797Z  * [new branch]              gh/swolchok/824/base        -> origin/gh/swolchok/824/base
2025-09-07T07:36:04.1438990Z  * [new branch]              gh/swolchok/824/head        -> origin/gh/swolchok/824/head
2025-09-07T07:36:04.1440163Z  * [new branch]              gh/swolchok/824/orig        -> origin/gh/swolchok/824/orig
2025-09-07T07:36:04.1441938Z  * [new branch]              gh/swolchok/825/base        -> origin/gh/swolchok/825/base
2025-09-07T07:36:04.1443181Z  * [new branch]              gh/swolchok/825/head        -> origin/gh/swolchok/825/head
2025-09-07T07:36:04.1444369Z  * [new branch]              gh/swolchok/825/orig        -> origin/gh/swolchok/825/orig
2025-09-07T07:36:04.1446199Z  * [new branch]              gh/swolchok/826/base        -> origin/gh/swolchok/826/base
2025-09-07T07:36:04.1447364Z  * [new branch]              gh/swolchok/826/head        -> origin/gh/swolchok/826/head
2025-09-07T07:36:04.1448448Z  * [new branch]              gh/swolchok/826/orig        -> origin/gh/swolchok/826/orig
2025-09-07T07:36:04.1450311Z  * [new branch]              gh/swolchok/827/base        -> origin/gh/swolchok/827/base
2025-09-07T07:36:04.1451571Z  * [new branch]              gh/swolchok/827/head        -> origin/gh/swolchok/827/head
2025-09-07T07:36:04.1452601Z  * [new branch]              gh/swolchok/827/orig        -> origin/gh/swolchok/827/orig
2025-09-07T07:36:04.1454427Z  * [new branch]              gh/swolchok/828/base        -> origin/gh/swolchok/828/base
2025-09-07T07:36:04.1455614Z  * [new branch]              gh/swolchok/828/head        -> origin/gh/swolchok/828/head
2025-09-07T07:36:04.1456792Z  * [new branch]              gh/swolchok/828/orig        -> origin/gh/swolchok/828/orig
2025-09-07T07:36:04.1458518Z  * [new branch]              gh/swolchok/829/base        -> origin/gh/swolchok/829/base
2025-09-07T07:36:04.1459726Z  * [new branch]              gh/swolchok/829/head        -> origin/gh/swolchok/829/head
2025-09-07T07:36:04.1460916Z  * [new branch]              gh/swolchok/829/orig        -> origin/gh/swolchok/829/orig
2025-09-07T07:36:04.1462906Z  * [new branch]              gh/swolchok/830/base        -> origin/gh/swolchok/830/base
2025-09-07T07:36:04.1464067Z  * [new branch]              gh/swolchok/830/head        -> origin/gh/swolchok/830/head
2025-09-07T07:36:04.1465214Z  * [new branch]              gh/swolchok/830/orig        -> origin/gh/swolchok/830/orig
2025-09-07T07:36:04.1466835Z  * [new branch]              gh/swolchok/831/base        -> origin/gh/swolchok/831/base
2025-09-07T07:36:04.1468140Z  * [new branch]              gh/swolchok/831/head        -> origin/gh/swolchok/831/head
2025-09-07T07:36:04.1469440Z  * [new branch]              gh/swolchok/831/orig        -> origin/gh/swolchok/831/orig
2025-09-07T07:36:04.1471868Z  * [new branch]              gh/swolchok/832/base        -> origin/gh/swolchok/832/base
2025-09-07T07:36:04.1473203Z  * [new branch]              gh/swolchok/832/head        -> origin/gh/swolchok/832/head
2025-09-07T07:36:04.1474395Z  * [new branch]              gh/swolchok/832/orig        -> origin/gh/swolchok/832/orig
2025-09-07T07:36:04.1476560Z  * [new branch]              gh/syed-ahmed/3/base        -> origin/gh/syed-ahmed/3/base
2025-09-07T07:36:04.1477733Z  * [new branch]              gh/syed-ahmed/3/head        -> origin/gh/syed-ahmed/3/head
2025-09-07T07:36:04.1478959Z  * [new branch]              gh/syed-ahmed/3/orig        -> origin/gh/syed-ahmed/3/orig
2025-09-07T07:36:04.1480723Z  * [new branch]              gh/syed-ahmed/4/base        -> origin/gh/syed-ahmed/4/base
2025-09-07T07:36:04.1481874Z  * [new branch]              gh/syed-ahmed/4/head        -> origin/gh/syed-ahmed/4/head
2025-09-07T07:36:04.1483061Z  * [new branch]              gh/syed-ahmed/4/orig        -> origin/gh/syed-ahmed/4/orig
2025-09-07T07:36:04.1484732Z  * [new branch]              gh/syed-ahmed/5/base        -> origin/gh/syed-ahmed/5/base
2025-09-07T07:36:04.1485915Z  * [new branch]              gh/syed-ahmed/5/head        -> origin/gh/syed-ahmed/5/head
2025-09-07T07:36:04.1487078Z  * [new branch]              gh/syed-ahmed/5/orig        -> origin/gh/syed-ahmed/5/orig
2025-09-07T07:36:04.1489379Z  * [new branch]              gh/teja-rao/4/base          -> origin/gh/teja-rao/4/base
2025-09-07T07:36:04.1490609Z  * [new branch]              gh/teja-rao/4/head          -> origin/gh/teja-rao/4/head
2025-09-07T07:36:04.1491817Z  * [new branch]              gh/teja-rao/4/orig          -> origin/gh/teja-rao/4/orig
2025-09-07T07:36:04.1493909Z  * [new branch]              gh/tianyu-l/2/base          -> origin/gh/tianyu-l/2/base
2025-09-07T07:36:04.1495116Z  * [new branch]              gh/tianyu-l/2/head          -> origin/gh/tianyu-l/2/head
2025-09-07T07:36:04.1496308Z  * [new branch]              gh/tianyu-l/2/orig          -> origin/gh/tianyu-l/2/orig
2025-09-07T07:36:04.1498092Z  * [new branch]              gh/tianyu-l/3/base          -> origin/gh/tianyu-l/3/base
2025-09-07T07:36:04.1499439Z  * [new branch]              gh/tianyu-l/3/head          -> origin/gh/tianyu-l/3/head
2025-09-07T07:36:04.1500656Z  * [new branch]              gh/tianyu-l/3/orig          -> origin/gh/tianyu-l/3/orig
2025-09-07T07:36:04.1502405Z  * [new branch]              gh/tianyu-l/4/base          -> origin/gh/tianyu-l/4/base
2025-09-07T07:36:04.1503931Z  * [new branch]              gh/tianyu-l/4/head          -> origin/gh/tianyu-l/4/head
2025-09-07T07:36:04.1505055Z  * [new branch]              gh/tianyu-l/4/orig          -> origin/gh/tianyu-l/4/orig
2025-09-07T07:36:04.1507284Z  * [new branch]              gh/tugsbayasgalan/1/base    -> origin/gh/tugsbayasgalan/1/base
2025-09-07T07:36:04.1508425Z  * [new branch]              gh/tugsbayasgalan/1/head    -> origin/gh/tugsbayasgalan/1/head
2025-09-07T07:36:04.1509751Z  * [new branch]              gh/tugsbayasgalan/1/orig    -> origin/gh/tugsbayasgalan/1/orig
2025-09-07T07:36:04.1511699Z  * [new branch]              gh/tugsbayasgalan/10/base   -> origin/gh/tugsbayasgalan/10/base
2025-09-07T07:36:04.1512896Z  * [new branch]              gh/tugsbayasgalan/10/head   -> origin/gh/tugsbayasgalan/10/head
2025-09-07T07:36:04.1514074Z  * [new branch]              gh/tugsbayasgalan/10/orig   -> origin/gh/tugsbayasgalan/10/orig
2025-09-07T07:36:04.1515749Z  * [new branch]              gh/tugsbayasgalan/11/base   -> origin/gh/tugsbayasgalan/11/base
2025-09-07T07:36:04.1516979Z  * [new branch]              gh/tugsbayasgalan/11/head   -> origin/gh/tugsbayasgalan/11/head
2025-09-07T07:36:04.1518145Z  * [new branch]              gh/tugsbayasgalan/11/orig   -> origin/gh/tugsbayasgalan/11/orig
2025-09-07T07:36:04.1519914Z  * [new branch]              gh/tugsbayasgalan/12/base   -> origin/gh/tugsbayasgalan/12/base
2025-09-07T07:36:04.1521116Z  * [new branch]              gh/tugsbayasgalan/12/head   -> origin/gh/tugsbayasgalan/12/head
2025-09-07T07:36:04.1522316Z  * [new branch]              gh/tugsbayasgalan/12/orig   -> origin/gh/tugsbayasgalan/12/orig
2025-09-07T07:36:04.1524117Z  * [new branch]              gh/tugsbayasgalan/13/base   -> origin/gh/tugsbayasgalan/13/base
2025-09-07T07:36:04.1525317Z  * [new branch]              gh/tugsbayasgalan/13/head   -> origin/gh/tugsbayasgalan/13/head
2025-09-07T07:36:04.1526554Z  * [new branch]              gh/tugsbayasgalan/13/orig   -> origin/gh/tugsbayasgalan/13/orig
2025-09-07T07:36:04.1528797Z  * [new branch]              gh/tugsbayasgalan/14/base   -> origin/gh/tugsbayasgalan/14/base
2025-09-07T07:36:04.1529948Z  * [new branch]              gh/tugsbayasgalan/14/head   -> origin/gh/tugsbayasgalan/14/head
2025-09-07T07:36:04.1531117Z  * [new branch]              gh/tugsbayasgalan/14/orig   -> origin/gh/tugsbayasgalan/14/orig
2025-09-07T07:36:04.1532983Z  * [new branch]              gh/tugsbayasgalan/15/base   -> origin/gh/tugsbayasgalan/15/base
2025-09-07T07:36:04.1534124Z  * [new branch]              gh/tugsbayasgalan/15/head   -> origin/gh/tugsbayasgalan/15/head
2025-09-07T07:36:04.1535435Z  * [new branch]              gh/tugsbayasgalan/15/orig   -> origin/gh/tugsbayasgalan/15/orig
2025-09-07T07:36:04.1537159Z  * [new branch]              gh/tugsbayasgalan/2/base    -> origin/gh/tugsbayasgalan/2/base
2025-09-07T07:36:04.1538534Z  * [new branch]              gh/tugsbayasgalan/2/head    -> origin/gh/tugsbayasgalan/2/head
2025-09-07T07:36:04.1539730Z  * [new branch]              gh/tugsbayasgalan/2/orig    -> origin/gh/tugsbayasgalan/2/orig
2025-09-07T07:36:04.1541730Z  * [new branch]              gh/tugsbayasgalan/3/base    -> origin/gh/tugsbayasgalan/3/base
2025-09-07T07:36:04.1543028Z  * [new branch]              gh/tugsbayasgalan/3/head    -> origin/gh/tugsbayasgalan/3/head
2025-09-07T07:36:04.1544222Z  * [new branch]              gh/tugsbayasgalan/3/orig    -> origin/gh/tugsbayasgalan/3/orig
2025-09-07T07:36:04.1545938Z  * [new branch]              gh/tugsbayasgalan/4/base    -> origin/gh/tugsbayasgalan/4/base
2025-09-07T07:36:04.1547327Z  * [new branch]              gh/tugsbayasgalan/4/head    -> origin/gh/tugsbayasgalan/4/head
2025-09-07T07:36:04.1548487Z  * [new branch]              gh/tugsbayasgalan/4/orig    -> origin/gh/tugsbayasgalan/4/orig
2025-09-07T07:36:04.1550271Z  * [new branch]              gh/tugsbayasgalan/5/base    -> origin/gh/tugsbayasgalan/5/base
2025-09-07T07:36:04.1551592Z  * [new branch]              gh/tugsbayasgalan/5/head    -> origin/gh/tugsbayasgalan/5/head
2025-09-07T07:36:04.1552719Z  * [new branch]              gh/tugsbayasgalan/5/orig    -> origin/gh/tugsbayasgalan/5/orig
2025-09-07T07:36:04.1554340Z  * [new branch]              gh/tugsbayasgalan/6/base    -> origin/gh/tugsbayasgalan/6/base
2025-09-07T07:36:04.1555644Z  * [new branch]              gh/tugsbayasgalan/6/head    -> origin/gh/tugsbayasgalan/6/head
2025-09-07T07:36:04.1556871Z  * [new branch]              gh/tugsbayasgalan/6/orig    -> origin/gh/tugsbayasgalan/6/orig
2025-09-07T07:36:04.1558621Z  * [new branch]              gh/tugsbayasgalan/7/base    -> origin/gh/tugsbayasgalan/7/base
2025-09-07T07:36:04.1559791Z  * [new branch]              gh/tugsbayasgalan/7/head    -> origin/gh/tugsbayasgalan/7/head
2025-09-07T07:36:04.1561056Z  * [new branch]              gh/tugsbayasgalan/7/orig    -> origin/gh/tugsbayasgalan/7/orig
2025-09-07T07:36:04.1562765Z  * [new branch]              gh/tugsbayasgalan/8/base    -> origin/gh/tugsbayasgalan/8/base
2025-09-07T07:36:04.1563899Z  * [new branch]              gh/tugsbayasgalan/8/head    -> origin/gh/tugsbayasgalan/8/head
2025-09-07T07:36:04.1565092Z  * [new branch]              gh/tugsbayasgalan/8/orig    -> origin/gh/tugsbayasgalan/8/orig
2025-09-07T07:36:04.1566743Z  * [new branch]              gh/tugsbayasgalan/9/base    -> origin/gh/tugsbayasgalan/9/base
2025-09-07T07:36:04.1567813Z  * [new branch]              gh/tugsbayasgalan/9/head    -> origin/gh/tugsbayasgalan/9/head
2025-09-07T07:36:04.1568973Z  * [new branch]              gh/tugsbayasgalan/9/orig    -> origin/gh/tugsbayasgalan/9/orig
2025-09-07T07:36:04.1571361Z  * [new branch]              gh/v0i0/1/base              -> origin/gh/v0i0/1/base
2025-09-07T07:36:04.1572540Z  * [new branch]              gh/v0i0/1/head              -> origin/gh/v0i0/1/head
2025-09-07T07:36:04.1573716Z  * [new branch]              gh/v0i0/1/orig              -> origin/gh/v0i0/1/orig
2025-09-07T07:36:04.1575422Z  * [new branch]              gh/v0i0/4/base              -> origin/gh/v0i0/4/base
2025-09-07T07:36:04.1576582Z  * [new branch]              gh/v0i0/4/head              -> origin/gh/v0i0/4/head
2025-09-07T07:36:04.1577758Z  * [new branch]              gh/v0i0/4/orig              -> origin/gh/v0i0/4/orig
2025-09-07T07:36:04.1579557Z  * [new branch]              gh/v0i0/6/base              -> origin/gh/v0i0/6/base
2025-09-07T07:36:04.1580755Z  * [new branch]              gh/v0i0/6/head              -> origin/gh/v0i0/6/head
2025-09-07T07:36:04.1581927Z  * [new branch]              gh/v0i0/6/orig              -> origin/gh/v0i0/6/orig
2025-09-07T07:36:04.1583659Z  * [new branch]              gh/v0i0/7/base              -> origin/gh/v0i0/7/base
2025-09-07T07:36:04.1584885Z  * [new branch]              gh/v0i0/7/head              -> origin/gh/v0i0/7/head
2025-09-07T07:36:04.1586056Z  * [new branch]              gh/v0i0/7/orig              -> origin/gh/v0i0/7/orig
2025-09-07T07:36:04.1587785Z  * [new branch]              gh/v0i0/8/base              -> origin/gh/v0i0/8/base
2025-09-07T07:36:04.1588911Z  * [new branch]              gh/v0i0/8/head              -> origin/gh/v0i0/8/head
2025-09-07T07:36:04.1590061Z  * [new branch]              gh/v0i0/8/orig              -> origin/gh/v0i0/8/orig
2025-09-07T07:36:04.1591788Z  * [new branch]              gh/v0i0/9/base              -> origin/gh/v0i0/9/base
2025-09-07T07:36:04.1592968Z  * [new branch]              gh/v0i0/9/head              -> origin/gh/v0i0/9/head
2025-09-07T07:36:04.1594265Z  * [new branch]              gh/v0i0/9/orig              -> origin/gh/v0i0/9/orig
2025-09-07T07:36:04.1596329Z  * [new branch]              gh/vkuzo/1/next             -> origin/gh/vkuzo/1/next
2025-09-07T07:36:04.1597991Z  * [new branch]              gh/vkuzo/2/next             -> origin/gh/vkuzo/2/next
2025-09-07T07:36:04.1599682Z  * [new branch]              gh/vkuzo/3/next             -> origin/gh/vkuzo/3/next
2025-09-07T07:36:04.1601372Z  * [new branch]              gh/vkuzo/4/base             -> origin/gh/vkuzo/4/base
2025-09-07T07:36:04.1602871Z  * [new branch]              gh/vkuzo/4/head             -> origin/gh/vkuzo/4/head
2025-09-07T07:36:04.1606374Z  * [new branch]              gh/vkuzo/4/orig             -> origin/gh/vkuzo/4/orig
2025-09-07T07:36:04.1608230Z  * [new branch]              gh/vkuzo/5/base             -> origin/gh/vkuzo/5/base
2025-09-07T07:36:04.1609614Z  * [new branch]              gh/vkuzo/5/head             -> origin/gh/vkuzo/5/head
2025-09-07T07:36:04.1611046Z  * [new branch]              gh/vkuzo/5/orig             -> origin/gh/vkuzo/5/orig
2025-09-07T07:36:04.1612912Z  * [new branch]              gh/vkuzo/6/base             -> origin/gh/vkuzo/6/base
2025-09-07T07:36:04.1614040Z  * [new branch]              gh/vkuzo/6/head             -> origin/gh/vkuzo/6/head
2025-09-07T07:36:04.1615300Z  * [new branch]              gh/vkuzo/6/orig             -> origin/gh/vkuzo/6/orig
2025-09-07T07:36:04.1616886Z  * [new branch]              gh/vkuzo/7/base             -> origin/gh/vkuzo/7/base
2025-09-07T07:36:04.1618410Z  * [new branch]              gh/vkuzo/7/head             -> origin/gh/vkuzo/7/head
2025-09-07T07:36:04.1619546Z  * [new branch]              gh/vkuzo/7/orig             -> origin/gh/vkuzo/7/orig
2025-09-07T07:36:04.1621772Z  * [new branch]              gh/wconstab/419/base        -> origin/gh/wconstab/419/base
2025-09-07T07:36:04.1622895Z  * [new branch]              gh/wconstab/419/head        -> origin/gh/wconstab/419/head
2025-09-07T07:36:04.1624103Z  * [new branch]              gh/wconstab/419/orig        -> origin/gh/wconstab/419/orig
2025-09-07T07:36:04.1625900Z  * [new branch]              gh/wconstab/424/base        -> origin/gh/wconstab/424/base
2025-09-07T07:36:04.1627021Z  * [new branch]              gh/wconstab/424/head        -> origin/gh/wconstab/424/head
2025-09-07T07:36:04.1628164Z  * [new branch]              gh/wconstab/424/orig        -> origin/gh/wconstab/424/orig
2025-09-07T07:36:04.1629873Z  * [new branch]              gh/wconstab/435/base        -> origin/gh/wconstab/435/base
2025-09-07T07:36:04.1631080Z  * [new branch]              gh/wconstab/435/head        -> origin/gh/wconstab/435/head
2025-09-07T07:36:04.1632272Z  * [new branch]              gh/wconstab/435/orig        -> origin/gh/wconstab/435/orig
2025-09-07T07:36:04.1633976Z  * [new branch]              gh/wconstab/438/base        -> origin/gh/wconstab/438/base
2025-09-07T07:36:04.1635201Z  * [new branch]              gh/wconstab/438/head        -> origin/gh/wconstab/438/head
2025-09-07T07:36:04.1636794Z  * [new branch]              gh/wconstab/438/orig        -> origin/gh/wconstab/438/orig
2025-09-07T07:36:04.1639091Z  * [new branch]              gh/wconstab/440/base        -> origin/gh/wconstab/440/base
2025-09-07T07:36:04.1640411Z  * [new branch]              gh/wconstab/440/head        -> origin/gh/wconstab/440/head
2025-09-07T07:36:04.1641677Z  * [new branch]              gh/wconstab/440/orig        -> origin/gh/wconstab/440/orig
2025-09-07T07:36:04.1643580Z  * [new branch]              gh/wconstab/441/base        -> origin/gh/wconstab/441/base
2025-09-07T07:36:04.1644721Z  * [new branch]              gh/wconstab/441/head        -> origin/gh/wconstab/441/head
2025-09-07T07:36:04.1645953Z  * [new branch]              gh/wconstab/441/orig        -> origin/gh/wconstab/441/orig
2025-09-07T07:36:04.1647925Z  * [new branch]              gh/wconstab/442/base        -> origin/gh/wconstab/442/base
2025-09-07T07:36:04.1649228Z  * [new branch]              gh/wconstab/442/head        -> origin/gh/wconstab/442/head
2025-09-07T07:36:04.1650438Z  * [new branch]              gh/wconstab/442/orig        -> origin/gh/wconstab/442/orig
2025-09-07T07:36:04.1652178Z  * [new branch]              gh/wconstab/443/base        -> origin/gh/wconstab/443/base
2025-09-07T07:36:04.1653382Z  * [new branch]              gh/wconstab/443/head        -> origin/gh/wconstab/443/head
2025-09-07T07:36:04.1654561Z  * [new branch]              gh/wconstab/443/orig        -> origin/gh/wconstab/443/orig
2025-09-07T07:36:04.1656430Z  * [new branch]              gh/wconstab/444/base        -> origin/gh/wconstab/444/base
2025-09-07T07:36:04.1657525Z  * [new branch]              gh/wconstab/444/head        -> origin/gh/wconstab/444/head
2025-09-07T07:36:04.1658860Z  * [new branch]              gh/wconstab/444/orig        -> origin/gh/wconstab/444/orig
2025-09-07T07:36:04.1660545Z  * [new branch]              gh/wconstab/445/base        -> origin/gh/wconstab/445/base
2025-09-07T07:36:04.1661723Z  * [new branch]              gh/wconstab/445/head        -> origin/gh/wconstab/445/head
2025-09-07T07:36:04.1662913Z  * [new branch]              gh/wconstab/445/orig        -> origin/gh/wconstab/445/orig
2025-09-07T07:36:04.1665035Z  * [new branch]              gh/wconstab/446/base        -> origin/gh/wconstab/446/base
2025-09-07T07:36:04.1666347Z  * [new branch]              gh/wconstab/446/head        -> origin/gh/wconstab/446/head
2025-09-07T07:36:04.1667814Z  * [new branch]              gh/wconstab/446/orig        -> origin/gh/wconstab/446/orig
2025-09-07T07:36:04.1669553Z  * [new branch]              gh/wconstab/447/base        -> origin/gh/wconstab/447/base
2025-09-07T07:36:04.1670714Z  * [new branch]              gh/wconstab/447/head        -> origin/gh/wconstab/447/head
2025-09-07T07:36:04.1671926Z  * [new branch]              gh/wconstab/447/orig        -> origin/gh/wconstab/447/orig
2025-09-07T07:36:04.1674222Z  * [new branch]              gh/weifengpy/27/base        -> origin/gh/weifengpy/27/base
2025-09-07T07:36:04.1675417Z  * [new branch]              gh/weifengpy/27/head        -> origin/gh/weifengpy/27/head
2025-09-07T07:36:04.1676590Z  * [new branch]              gh/weifengpy/27/orig        -> origin/gh/weifengpy/27/orig
2025-09-07T07:36:04.1678355Z  * [new branch]              gh/weifengpy/30/base        -> origin/gh/weifengpy/30/base
2025-09-07T07:36:04.1679514Z  * [new branch]              gh/weifengpy/30/head        -> origin/gh/weifengpy/30/head
2025-09-07T07:36:04.1680685Z  * [new branch]              gh/weifengpy/30/orig        -> origin/gh/weifengpy/30/orig
2025-09-07T07:36:04.1683278Z  * [new branch]              gh/williamwen42/196/base    -> origin/gh/williamwen42/196/base
2025-09-07T07:36:04.1684635Z  * [new branch]              gh/williamwen42/196/head    -> origin/gh/williamwen42/196/head
2025-09-07T07:36:04.1685918Z  * [new branch]              gh/williamwen42/196/orig    -> origin/gh/williamwen42/196/orig
2025-09-07T07:36:04.1687660Z  * [new branch]              gh/williamwen42/250/base    -> origin/gh/williamwen42/250/base
2025-09-07T07:36:04.1688882Z  * [new branch]              gh/williamwen42/250/head    -> origin/gh/williamwen42/250/head
2025-09-07T07:36:04.1690110Z  * [new branch]              gh/williamwen42/250/orig    -> origin/gh/williamwen42/250/orig
2025-09-07T07:36:04.1691944Z  * [new branch]              gh/williamwen42/258/base    -> origin/gh/williamwen42/258/base
2025-09-07T07:36:04.1693140Z  * [new branch]              gh/williamwen42/258/head    -> origin/gh/williamwen42/258/head
2025-09-07T07:36:04.1694364Z  * [new branch]              gh/williamwen42/258/orig    -> origin/gh/williamwen42/258/orig
2025-09-07T07:36:04.1696037Z  * [new branch]              gh/williamwen42/266/base    -> origin/gh/williamwen42/266/base
2025-09-07T07:36:04.1697305Z  * [new branch]              gh/williamwen42/266/head    -> origin/gh/williamwen42/266/head
2025-09-07T07:36:04.1698644Z  * [new branch]              gh/williamwen42/266/orig    -> origin/gh/williamwen42/266/orig
2025-09-07T07:36:04.1700378Z  * [new branch]              gh/williamwen42/267/base    -> origin/gh/williamwen42/267/base
2025-09-07T07:36:04.1701633Z  * [new branch]              gh/williamwen42/267/head    -> origin/gh/williamwen42/267/head
2025-09-07T07:36:04.1702794Z  * [new branch]              gh/williamwen42/267/orig    -> origin/gh/williamwen42/267/orig
2025-09-07T07:36:04.1705779Z  * [new branch]              gh/williamwen42/270/base    -> origin/gh/williamwen42/270/base
2025-09-07T07:36:04.1706134Z  * [new branch]              gh/williamwen42/270/head    -> origin/gh/williamwen42/270/head
2025-09-07T07:36:04.1707391Z  * [new branch]              gh/williamwen42/270/orig    -> origin/gh/williamwen42/270/orig
2025-09-07T07:36:04.1708991Z  * [new branch]              gh/williamwen42/271/base    -> origin/gh/williamwen42/271/base
2025-09-07T07:36:04.1710224Z  * [new branch]              gh/williamwen42/271/head    -> origin/gh/williamwen42/271/head
2025-09-07T07:36:04.1711443Z  * [new branch]              gh/williamwen42/271/orig    -> origin/gh/williamwen42/271/orig
2025-09-07T07:36:04.1713164Z  * [new branch]              gh/williamwen42/272/base    -> origin/gh/williamwen42/272/base
2025-09-07T07:36:04.1714372Z  * [new branch]              gh/williamwen42/272/head    -> origin/gh/williamwen42/272/head
2025-09-07T07:36:04.1715601Z  * [new branch]              gh/williamwen42/272/orig    -> origin/gh/williamwen42/272/orig
2025-09-07T07:36:04.1717306Z  * [new branch]              gh/williamwen42/274/base    -> origin/gh/williamwen42/274/base
2025-09-07T07:36:04.1718575Z  * [new branch]              gh/williamwen42/274/head    -> origin/gh/williamwen42/274/head
2025-09-07T07:36:04.1720023Z  * [new branch]              gh/williamwen42/274/orig    -> origin/gh/williamwen42/274/orig
2025-09-07T07:36:04.1721702Z  * [new branch]              gh/williamwen42/275/base    -> origin/gh/williamwen42/275/base
2025-09-07T07:36:04.1722868Z  * [new branch]              gh/williamwen42/275/head    -> origin/gh/williamwen42/275/head
2025-09-07T07:36:04.1724532Z  * [new branch]              gh/williamwen42/276/base    -> origin/gh/williamwen42/276/base
2025-09-07T07:36:04.1725739Z  * [new branch]              gh/williamwen42/276/head    -> origin/gh/williamwen42/276/head
2025-09-07T07:36:04.1726931Z  * [new branch]              gh/williamwen42/276/orig    -> origin/gh/williamwen42/276/orig
2025-09-07T07:36:04.1728712Z  * [new branch]              gh/williamwen42/277/base    -> origin/gh/williamwen42/277/base
2025-09-07T07:36:04.1729899Z  * [new branch]              gh/williamwen42/277/head    -> origin/gh/williamwen42/277/head
2025-09-07T07:36:04.1731087Z  * [new branch]              gh/williamwen42/277/orig    -> origin/gh/williamwen42/277/orig
2025-09-07T07:36:04.1732870Z  * [new branch]              gh/williamwen42/278/base    -> origin/gh/williamwen42/278/base
2025-09-07T07:36:04.1734048Z  * [new branch]              gh/williamwen42/278/head    -> origin/gh/williamwen42/278/head
2025-09-07T07:36:04.1735242Z  * [new branch]              gh/williamwen42/278/orig    -> origin/gh/williamwen42/278/orig
2025-09-07T07:36:04.1736961Z  * [new branch]              gh/williamwen42/279/base    -> origin/gh/williamwen42/279/base
2025-09-07T07:36:04.1738209Z  * [new branch]              gh/williamwen42/279/head    -> origin/gh/williamwen42/279/head
2025-09-07T07:36:04.1739426Z  * [new branch]              gh/williamwen42/279/orig    -> origin/gh/williamwen42/279/orig
2025-09-07T07:36:04.1741348Z  * [new branch]              gh/williamwen42/280/base    -> origin/gh/williamwen42/280/base
2025-09-07T07:36:04.1744987Z  * [new branch]              gh/williamwen42/280/head    -> origin/gh/williamwen42/280/head
2025-09-07T07:36:04.1746737Z  * [new branch]              gh/williamwen42/280/orig    -> origin/gh/williamwen42/280/orig
2025-09-07T07:36:04.1746889Z  * [new branch]              gh/williamwen42/281/base    -> origin/gh/williamwen42/281/base
2025-09-07T07:36:04.1747027Z  * [new branch]              gh/williamwen42/281/head    -> origin/gh/williamwen42/281/head
2025-09-07T07:36:04.1747648Z  * [new branch]              gh/williamwen42/281/orig    -> origin/gh/williamwen42/281/orig
2025-09-07T07:36:04.1749692Z  * [new branch]              gh/williamwen42/282/base    -> origin/gh/williamwen42/282/base
2025-09-07T07:36:04.1750919Z  * [new branch]              gh/williamwen42/282/head    -> origin/gh/williamwen42/282/head
2025-09-07T07:36:04.1752079Z  * [new branch]              gh/williamwen42/282/orig    -> origin/gh/williamwen42/282/orig
2025-09-07T07:36:04.1754082Z  * [new branch]              gh/williamwen42/283/base    -> origin/gh/williamwen42/283/base
2025-09-07T07:36:04.1755643Z  * [new branch]              gh/williamwen42/283/head    -> origin/gh/williamwen42/283/head
2025-09-07T07:36:04.1756783Z  * [new branch]              gh/williamwen42/283/orig    -> origin/gh/williamwen42/283/orig
2025-09-07T07:36:04.1758757Z  * [new branch]              gh/williamwen42/284/base    -> origin/gh/williamwen42/284/base
2025-09-07T07:36:04.1759909Z  * [new branch]              gh/williamwen42/284/head    -> origin/gh/williamwen42/284/head
2025-09-07T07:36:04.1761074Z  * [new branch]              gh/williamwen42/284/orig    -> origin/gh/williamwen42/284/orig
2025-09-07T07:36:04.1762664Z  * [new branch]              gh/williamwen42/285/base    -> origin/gh/williamwen42/285/base
2025-09-07T07:36:04.1763916Z  * [new branch]              gh/williamwen42/285/head    -> origin/gh/williamwen42/285/head
2025-09-07T07:36:04.1765094Z  * [new branch]              gh/williamwen42/285/orig    -> origin/gh/williamwen42/285/orig
2025-09-07T07:36:04.1766717Z  * [new branch]              gh/williamwen42/286/base    -> origin/gh/williamwen42/286/base
2025-09-07T07:36:04.1767876Z  * [new branch]              gh/williamwen42/286/head    -> origin/gh/williamwen42/286/head
2025-09-07T07:36:04.1769048Z  * [new branch]              gh/williamwen42/286/orig    -> origin/gh/williamwen42/286/orig
2025-09-07T07:36:04.1770833Z  * [new branch]              gh/williamwen42/287/base    -> origin/gh/williamwen42/287/base
2025-09-07T07:36:04.1772023Z  * [new branch]              gh/williamwen42/287/head    -> origin/gh/williamwen42/287/head
2025-09-07T07:36:04.1773403Z  * [new branch]              gh/williamwen42/287/orig    -> origin/gh/williamwen42/287/orig
2025-09-07T07:36:04.1775290Z  * [new branch]              gh/williamwen42/288/base    -> origin/gh/williamwen42/288/base
2025-09-07T07:36:04.1776527Z  * [new branch]              gh/williamwen42/288/head    -> origin/gh/williamwen42/288/head
2025-09-07T07:36:04.1777755Z  * [new branch]              gh/williamwen42/288/orig    -> origin/gh/williamwen42/288/orig
2025-09-07T07:36:04.1779537Z  * [new branch]              gh/williamwen42/289/base    -> origin/gh/williamwen42/289/base
2025-09-07T07:36:04.1780719Z  * [new branch]              gh/williamwen42/289/head    -> origin/gh/williamwen42/289/head
2025-09-07T07:36:04.1781896Z  * [new branch]              gh/williamwen42/289/orig    -> origin/gh/williamwen42/289/orig
2025-09-07T07:36:04.1784196Z  * [new branch]              gh/wychi/1/base             -> origin/gh/wychi/1/base
2025-09-07T07:36:04.1785440Z  * [new branch]              gh/wychi/1/head             -> origin/gh/wychi/1/head
2025-09-07T07:36:04.1786662Z  * [new branch]              gh/wychi/1/orig             -> origin/gh/wychi/1/orig
2025-09-07T07:36:04.1788772Z  * [new branch]              gh/xmfan/169/base           -> origin/gh/xmfan/169/base
2025-09-07T07:36:04.1790038Z  * [new branch]              gh/xmfan/169/head           -> origin/gh/xmfan/169/head
2025-09-07T07:36:04.1791771Z  * [new branch]              gh/xmfan/170/base           -> origin/gh/xmfan/170/base
2025-09-07T07:36:04.1792960Z  * [new branch]              gh/xmfan/170/head           -> origin/gh/xmfan/170/head
2025-09-07T07:36:04.1794753Z  * [new branch]              gh/xmfan/18/base            -> origin/gh/xmfan/18/base
2025-09-07T07:36:04.1795922Z  * [new branch]              gh/xmfan/18/head            -> origin/gh/xmfan/18/head
2025-09-07T07:36:04.1797571Z  * [new branch]              gh/xmfan/229/base           -> origin/gh/xmfan/229/base
2025-09-07T07:36:04.1798749Z  * [new branch]              gh/xmfan/229/head           -> origin/gh/xmfan/229/head
2025-09-07T07:36:04.1799904Z  * [new branch]              gh/xmfan/229/orig           -> origin/gh/xmfan/229/orig
2025-09-07T07:36:04.1801580Z  * [new branch]              gh/xmfan/237/base           -> origin/gh/xmfan/237/base
2025-09-07T07:36:04.1802861Z  * [new branch]              gh/xmfan/237/head           -> origin/gh/xmfan/237/head
2025-09-07T07:36:04.1804364Z  * [new branch]              gh/xmfan/237/orig           -> origin/gh/xmfan/237/orig
2025-09-07T07:36:04.1806111Z  * [new branch]              gh/xmfan/244/base           -> origin/gh/xmfan/244/base
2025-09-07T07:36:04.1807270Z  * [new branch]              gh/xmfan/244/head           -> origin/gh/xmfan/244/head
2025-09-07T07:36:04.1808443Z  * [new branch]              gh/xmfan/244/orig           -> origin/gh/xmfan/244/orig
2025-09-07T07:36:04.1810125Z  * [new branch]              gh/xmfan/246/base           -> origin/gh/xmfan/246/base
2025-09-07T07:36:04.1811327Z  * [new branch]              gh/xmfan/246/head           -> origin/gh/xmfan/246/head
2025-09-07T07:36:04.1812491Z  * [new branch]              gh/xmfan/246/orig           -> origin/gh/xmfan/246/orig
2025-09-07T07:36:04.1814202Z  * [new branch]              gh/xmfan/253/base           -> origin/gh/xmfan/253/base
2025-09-07T07:36:04.1815405Z  * [new branch]              gh/xmfan/253/head           -> origin/gh/xmfan/253/head
2025-09-07T07:36:04.1816591Z  * [new branch]              gh/xmfan/253/orig           -> origin/gh/xmfan/253/orig
2025-09-07T07:36:04.1818338Z  * [new branch]              gh/xmfan/254/base           -> origin/gh/xmfan/254/base
2025-09-07T07:36:04.1819518Z  * [new branch]              gh/xmfan/254/head           -> origin/gh/xmfan/254/head
2025-09-07T07:36:04.1820706Z  * [new branch]              gh/xmfan/254/orig           -> origin/gh/xmfan/254/orig
2025-09-07T07:36:04.1822510Z  * [new branch]              gh/xmfan/260/base           -> origin/gh/xmfan/260/base
2025-09-07T07:36:04.1823708Z  * [new branch]              gh/xmfan/260/head           -> origin/gh/xmfan/260/head
2025-09-07T07:36:04.1824846Z  * [new branch]              gh/xmfan/260/orig           -> origin/gh/xmfan/260/orig
2025-09-07T07:36:04.1826543Z  * [new branch]              gh/xmfan/262/base           -> origin/gh/xmfan/262/base
2025-09-07T07:36:04.1827879Z  * [new branch]              gh/xmfan/262/head           -> origin/gh/xmfan/262/head
2025-09-07T07:36:04.1829073Z  * [new branch]              gh/xmfan/262/orig           -> origin/gh/xmfan/262/orig
2025-09-07T07:36:04.1830780Z  * [new branch]              gh/xmfan/263/base           -> origin/gh/xmfan/263/base
2025-09-07T07:36:04.1831980Z  * [new branch]              gh/xmfan/263/head           -> origin/gh/xmfan/263/head
2025-09-07T07:36:04.1833211Z  * [new branch]              gh/xmfan/263/orig           -> origin/gh/xmfan/263/orig
2025-09-07T07:36:04.1834883Z  * [new branch]              gh/xmfan/264/base           -> origin/gh/xmfan/264/base
2025-09-07T07:36:04.1836085Z  * [new branch]              gh/xmfan/264/head           -> origin/gh/xmfan/264/head
2025-09-07T07:36:04.1837272Z  * [new branch]              gh/xmfan/264/orig           -> origin/gh/xmfan/264/orig
2025-09-07T07:36:04.1838998Z  * [new branch]              gh/xmfan/274/base           -> origin/gh/xmfan/274/base
2025-09-07T07:36:04.1840162Z  * [new branch]              gh/xmfan/274/head           -> origin/gh/xmfan/274/head
2025-09-07T07:36:04.1841404Z  * [new branch]              gh/xmfan/274/orig           -> origin/gh/xmfan/274/orig
2025-09-07T07:36:04.1843142Z  * [new branch]              gh/xmfan/276/base           -> origin/gh/xmfan/276/base
2025-09-07T07:36:04.1844760Z  * [new branch]              gh/xmfan/276/head           -> origin/gh/xmfan/276/head
2025-09-07T07:36:04.1846009Z  * [new branch]              gh/xmfan/276/orig           -> origin/gh/xmfan/276/orig
2025-09-07T07:36:04.1847674Z  * [new branch]              gh/xmfan/277/base           -> origin/gh/xmfan/277/base
2025-09-07T07:36:04.1848861Z  * [new branch]              gh/xmfan/277/head           -> origin/gh/xmfan/277/head
2025-09-07T07:36:04.1850023Z  * [new branch]              gh/xmfan/277/orig           -> origin/gh/xmfan/277/orig
2025-09-07T07:36:04.1851720Z  * [new branch]              gh/xmfan/278/base           -> origin/gh/xmfan/278/base
2025-09-07T07:36:04.1852884Z  * [new branch]              gh/xmfan/278/head           -> origin/gh/xmfan/278/head
2025-09-07T07:36:04.1854186Z  * [new branch]              gh/xmfan/278/orig           -> origin/gh/xmfan/278/orig
2025-09-07T07:36:04.1856221Z  * [new branch]              gh/xmfan/279/base           -> origin/gh/xmfan/279/base
2025-09-07T07:36:04.1857407Z  * [new branch]              gh/xmfan/279/head           -> origin/gh/xmfan/279/head
2025-09-07T07:36:04.1858661Z  * [new branch]              gh/xmfan/279/orig           -> origin/gh/xmfan/279/orig
2025-09-07T07:36:04.1860483Z  * [new branch]              gh/xmfan/280/base           -> origin/gh/xmfan/280/base
2025-09-07T07:36:04.1861635Z  * [new branch]              gh/xmfan/280/head           -> origin/gh/xmfan/280/head
2025-09-07T07:36:04.1862941Z  * [new branch]              gh/xmfan/280/orig           -> origin/gh/xmfan/280/orig
2025-09-07T07:36:04.1864675Z  * [new branch]              gh/xmfan/281/base           -> origin/gh/xmfan/281/base
2025-09-07T07:36:04.1865870Z  * [new branch]              gh/xmfan/281/head           -> origin/gh/xmfan/281/head
2025-09-07T07:36:04.1867042Z  * [new branch]              gh/xmfan/281/orig           -> origin/gh/xmfan/281/orig
2025-09-07T07:36:04.1868757Z  * [new branch]              gh/xmfan/282/base           -> origin/gh/xmfan/282/base
2025-09-07T07:36:04.1870015Z  * [new branch]              gh/xmfan/282/head           -> origin/gh/xmfan/282/head
2025-09-07T07:36:04.1871765Z  * [new branch]              gh/xmfan/283/base           -> origin/gh/xmfan/283/base
2025-09-07T07:36:04.1873029Z  * [new branch]              gh/xmfan/283/head           -> origin/gh/xmfan/283/head
2025-09-07T07:36:04.1874272Z  * [new branch]              gh/xmfan/283/orig           -> origin/gh/xmfan/283/orig
2025-09-07T07:36:04.1876436Z  * [new branch]              gh/xuanzhang816/14/base     -> origin/gh/xuanzhang816/14/base
2025-09-07T07:36:04.1880494Z  * [new branch]              gh/xuanzhang816/14/head     -> origin/gh/xuanzhang816/14/head
2025-09-07T07:36:04.1881739Z  * [new branch]              gh/xuanzhang816/14/orig     -> origin/gh/xuanzhang816/14/orig
2025-09-07T07:36:04.1883423Z  * [new branch]              gh/xuanzhang816/19/base     -> origin/gh/xuanzhang816/19/base
2025-09-07T07:36:04.1884589Z  * [new branch]              gh/xuanzhang816/19/head     -> origin/gh/xuanzhang816/19/head
2025-09-07T07:36:04.1885826Z  * [new branch]              gh/xuanzhang816/19/orig     -> origin/gh/xuanzhang816/19/orig
2025-09-07T07:36:04.1887537Z  * [new branch]              gh/xuanzhang816/22/base     -> origin/gh/xuanzhang816/22/base
2025-09-07T07:36:04.1888732Z  * [new branch]              gh/xuanzhang816/22/head     -> origin/gh/xuanzhang816/22/head
2025-09-07T07:36:04.1889915Z  * [new branch]              gh/xuanzhang816/22/orig     -> origin/gh/xuanzhang816/22/orig
2025-09-07T07:36:04.1891701Z  * [new branch]              gh/xuanzhang816/23/base     -> origin/gh/xuanzhang816/23/base
2025-09-07T07:36:04.1892909Z  * [new branch]              gh/xuanzhang816/23/head     -> origin/gh/xuanzhang816/23/head
2025-09-07T07:36:04.1894089Z  * [new branch]              gh/xuanzhang816/23/orig     -> origin/gh/xuanzhang816/23/orig
2025-09-07T07:36:04.1895756Z  * [new branch]              gh/xuanzhang816/24/base     -> origin/gh/xuanzhang816/24/base
2025-09-07T07:36:04.1896933Z  * [new branch]              gh/xuanzhang816/24/head     -> origin/gh/xuanzhang816/24/head
2025-09-07T07:36:04.1898565Z  * [new branch]              gh/xuanzhang816/24/orig     -> origin/gh/xuanzhang816/24/orig
2025-09-07T07:36:04.1900273Z  * [new branch]              gh/xuanzhang816/25/base     -> origin/gh/xuanzhang816/25/base
2025-09-07T07:36:04.1901648Z  * [new branch]              gh/xuanzhang816/25/head     -> origin/gh/xuanzhang816/25/head
2025-09-07T07:36:04.1902826Z  * [new branch]              gh/xuanzhang816/25/orig     -> origin/gh/xuanzhang816/25/orig
2025-09-07T07:36:04.1905138Z  * [new branch]              gh/xuanzhang816/26/base     -> origin/gh/xuanzhang816/26/base
2025-09-07T07:36:04.1906337Z  * [new branch]              gh/xuanzhang816/26/head     -> origin/gh/xuanzhang816/26/head
2025-09-07T07:36:04.1907647Z  * [new branch]              gh/xuanzhang816/26/orig     -> origin/gh/xuanzhang816/26/orig
2025-09-07T07:36:04.1909853Z  * [new branch]              gh/yanbing-j/11/base        -> origin/gh/yanbing-j/11/base
2025-09-07T07:36:04.1911048Z  * [new branch]              gh/yanbing-j/11/head        -> origin/gh/yanbing-j/11/head
2025-09-07T07:36:04.1912254Z  * [new branch]              gh/yanbing-j/11/orig        -> origin/gh/yanbing-j/11/orig
2025-09-07T07:36:04.1914053Z  * [new branch]              gh/yanbing-j/12/base        -> origin/gh/yanbing-j/12/base
2025-09-07T07:36:04.1915666Z  * [new branch]              gh/yanbing-j/12/head        -> origin/gh/yanbing-j/12/head
2025-09-07T07:36:04.1916903Z  * [new branch]              gh/yanbing-j/12/orig        -> origin/gh/yanbing-j/12/orig
2025-09-07T07:36:04.1918630Z  * [new branch]              gh/yanbing-j/13/base        -> origin/gh/yanbing-j/13/base
2025-09-07T07:36:04.1919828Z  * [new branch]              gh/yanbing-j/13/head        -> origin/gh/yanbing-j/13/head
2025-09-07T07:36:04.1921017Z  * [new branch]              gh/yanbing-j/13/orig        -> origin/gh/yanbing-j/13/orig
2025-09-07T07:36:04.1922778Z  * [new branch]              gh/yanbing-j/14/base        -> origin/gh/yanbing-j/14/base
2025-09-07T07:36:04.1924013Z  * [new branch]              gh/yanbing-j/14/head        -> origin/gh/yanbing-j/14/head
2025-09-07T07:36:04.1925219Z  * [new branch]              gh/yanbing-j/14/orig        -> origin/gh/yanbing-j/14/orig
2025-09-07T07:36:04.1926881Z  * [new branch]              gh/yanbing-j/15/base        -> origin/gh/yanbing-j/15/base
2025-09-07T07:36:04.1928074Z  * [new branch]              gh/yanbing-j/15/head        -> origin/gh/yanbing-j/15/head
2025-09-07T07:36:04.1929271Z  * [new branch]              gh/yanbing-j/15/orig        -> origin/gh/yanbing-j/15/orig
2025-09-07T07:36:04.1930929Z  * [new branch]              gh/yanbing-j/18/base        -> origin/gh/yanbing-j/18/base
2025-09-07T07:36:04.1932119Z  * [new branch]              gh/yanbing-j/18/head        -> origin/gh/yanbing-j/18/head
2025-09-07T07:36:04.1933304Z  * [new branch]              gh/yanbing-j/18/orig        -> origin/gh/yanbing-j/18/orig
2025-09-07T07:36:04.1935022Z  * [new branch]              gh/yanbing-j/19/base        -> origin/gh/yanbing-j/19/base
2025-09-07T07:36:04.1936231Z  * [new branch]              gh/yanbing-j/19/head        -> origin/gh/yanbing-j/19/head
2025-09-07T07:36:04.1937530Z  * [new branch]              gh/yanbing-j/19/orig        -> origin/gh/yanbing-j/19/orig
2025-09-07T07:36:04.1939407Z  * [new branch]              gh/yanbing-j/20/base        -> origin/gh/yanbing-j/20/base
2025-09-07T07:36:04.1940607Z  * [new branch]              gh/yanbing-j/20/head        -> origin/gh/yanbing-j/20/head
2025-09-07T07:36:04.1941847Z  * [new branch]              gh/yanbing-j/20/orig        -> origin/gh/yanbing-j/20/orig
2025-09-07T07:36:04.1943576Z  * [new branch]              gh/yanbing-j/21/base        -> origin/gh/yanbing-j/21/base
2025-09-07T07:36:04.1944797Z  * [new branch]              gh/yanbing-j/21/head        -> origin/gh/yanbing-j/21/head
2025-09-07T07:36:04.1946448Z  * [new branch]              gh/yanbing-j/22/base        -> origin/gh/yanbing-j/22/base
2025-09-07T07:36:04.1947649Z  * [new branch]              gh/yanbing-j/22/head        -> origin/gh/yanbing-j/22/head
2025-09-07T07:36:04.1948854Z  * [new branch]              gh/yanbing-j/22/orig        -> origin/gh/yanbing-j/22/orig
2025-09-07T07:36:04.1950619Z  * [new branch]              gh/yanbing-j/23/base        -> origin/gh/yanbing-j/23/base
2025-09-07T07:36:04.1951750Z  * [new branch]              gh/yanbing-j/23/head        -> origin/gh/yanbing-j/23/head
2025-09-07T07:36:04.1952917Z  * [new branch]              gh/yanbing-j/23/orig        -> origin/gh/yanbing-j/23/orig
2025-09-07T07:36:04.1954629Z  * [new branch]              gh/yanbing-j/24/base        -> origin/gh/yanbing-j/24/base
2025-09-07T07:36:04.1955818Z  * [new branch]              gh/yanbing-j/24/head        -> origin/gh/yanbing-j/24/head
2025-09-07T07:36:04.1957041Z  * [new branch]              gh/yanbing-j/24/orig        -> origin/gh/yanbing-j/24/orig
2025-09-07T07:36:04.1958721Z  * [new branch]              gh/yanbing-j/25/base        -> origin/gh/yanbing-j/25/base
2025-09-07T07:36:04.1959901Z  * [new branch]              gh/yanbing-j/25/head        -> origin/gh/yanbing-j/25/head
2025-09-07T07:36:04.1961079Z  * [new branch]              gh/yanbing-j/25/orig        -> origin/gh/yanbing-j/25/orig
2025-09-07T07:36:04.1962766Z  * [new branch]              gh/yanbing-j/26/base        -> origin/gh/yanbing-j/26/base
2025-09-07T07:36:04.1963924Z  * [new branch]              gh/yanbing-j/26/head        -> origin/gh/yanbing-j/26/head
2025-09-07T07:36:04.1965636Z  * [new branch]              gh/yanbing-j/26/orig        -> origin/gh/yanbing-j/26/orig
2025-09-07T07:36:04.1967350Z  * [new branch]              gh/yanbing-j/36/base        -> origin/gh/yanbing-j/36/base
2025-09-07T07:36:04.1968524Z  * [new branch]              gh/yanbing-j/36/head        -> origin/gh/yanbing-j/36/head
2025-09-07T07:36:04.1969718Z  * [new branch]              gh/yanbing-j/36/orig        -> origin/gh/yanbing-j/36/orig
2025-09-07T07:36:04.1971438Z  * [new branch]              gh/yanbing-j/37/base        -> origin/gh/yanbing-j/37/base
2025-09-07T07:36:04.1972776Z  * [new branch]              gh/yanbing-j/37/head        -> origin/gh/yanbing-j/37/head
2025-09-07T07:36:04.1973945Z  * [new branch]              gh/yanbing-j/37/orig        -> origin/gh/yanbing-j/37/orig
2025-09-07T07:36:04.1976151Z  * [new branch]              gh/yangw-dev/12/base        -> origin/gh/yangw-dev/12/base
2025-09-07T07:36:04.1977338Z  * [new branch]              gh/yangw-dev/12/head        -> origin/gh/yangw-dev/12/head
2025-09-07T07:36:04.1978620Z  * [new branch]              gh/yangw-dev/12/orig        -> origin/gh/yangw-dev/12/orig
2025-09-07T07:36:04.1980327Z  * [new branch]              gh/yangw-dev/13/base        -> origin/gh/yangw-dev/13/base
2025-09-07T07:36:04.1981545Z  * [new branch]              gh/yangw-dev/13/head        -> origin/gh/yangw-dev/13/head
2025-09-07T07:36:04.1982715Z  * [new branch]              gh/yangw-dev/13/orig        -> origin/gh/yangw-dev/13/orig
2025-09-07T07:36:04.1984840Z  * [new branch]              gh/yangw-dev/14/base        -> origin/gh/yangw-dev/14/base
2025-09-07T07:36:04.1986028Z  * [new branch]              gh/yangw-dev/14/head        -> origin/gh/yangw-dev/14/head
2025-09-07T07:36:04.1987249Z  * [new branch]              gh/yangw-dev/14/orig        -> origin/gh/yangw-dev/14/orig
2025-09-07T07:36:04.1988975Z  * [new branch]              gh/yangw-dev/15/base        -> origin/gh/yangw-dev/15/base
2025-09-07T07:36:04.1990164Z  * [new branch]              gh/yangw-dev/15/head        -> origin/gh/yangw-dev/15/head
2025-09-07T07:36:04.1991365Z  * [new branch]              gh/yangw-dev/15/orig        -> origin/gh/yangw-dev/15/orig
2025-09-07T07:36:04.1993117Z  * [new branch]              gh/yangw-dev/16/base        -> origin/gh/yangw-dev/16/base
2025-09-07T07:36:04.1994297Z  * [new branch]              gh/yangw-dev/16/head        -> origin/gh/yangw-dev/16/head
2025-09-07T07:36:04.1995443Z  * [new branch]              gh/yangw-dev/16/orig        -> origin/gh/yangw-dev/16/orig
2025-09-07T07:36:04.1997141Z  * [new branch]              gh/yangw-dev/17/base        -> origin/gh/yangw-dev/17/base
2025-09-07T07:36:04.1998335Z  * [new branch]              gh/yangw-dev/17/head        -> origin/gh/yangw-dev/17/head
2025-09-07T07:36:04.1999469Z  * [new branch]              gh/yangw-dev/17/orig        -> origin/gh/yangw-dev/17/orig
2025-09-07T07:36:04.2001130Z  * [new branch]              gh/yangw-dev/18/base        -> origin/gh/yangw-dev/18/base
2025-09-07T07:36:04.2002341Z  * [new branch]              gh/yangw-dev/18/head        -> origin/gh/yangw-dev/18/head
2025-09-07T07:36:04.2003736Z  * [new branch]              gh/yangw-dev/18/orig        -> origin/gh/yangw-dev/18/orig
2025-09-07T07:36:04.2005551Z  * [new branch]              gh/yangw-dev/19/base        -> origin/gh/yangw-dev/19/base
2025-09-07T07:36:04.2007055Z  * [new branch]              gh/yangw-dev/19/head        -> origin/gh/yangw-dev/19/head
2025-09-07T07:36:04.2007949Z  * [new branch]              gh/yangw-dev/19/orig        -> origin/gh/yangw-dev/19/orig
2025-09-07T07:36:04.2009729Z  * [new branch]              gh/yangw-dev/20/base        -> origin/gh/yangw-dev/20/base
2025-09-07T07:36:04.2010904Z  * [new branch]              gh/yangw-dev/20/head        -> origin/gh/yangw-dev/20/head
2025-09-07T07:36:04.2012489Z  * [new branch]              gh/yangw-dev/20/orig        -> origin/gh/yangw-dev/20/orig
2025-09-07T07:36:04.2014240Z  * [new branch]              gh/yangw-dev/21/base        -> origin/gh/yangw-dev/21/base
2025-09-07T07:36:04.2015445Z  * [new branch]              gh/yangw-dev/21/head        -> origin/gh/yangw-dev/21/head
2025-09-07T07:36:04.2016652Z  * [new branch]              gh/yangw-dev/21/orig        -> origin/gh/yangw-dev/21/orig
2025-09-07T07:36:04.2018471Z  * [new branch]              gh/yangw-dev/22/base        -> origin/gh/yangw-dev/22/base
2025-09-07T07:36:04.2019753Z  * [new branch]              gh/yangw-dev/22/head        -> origin/gh/yangw-dev/22/head
2025-09-07T07:36:04.2020916Z  * [new branch]              gh/yangw-dev/22/orig        -> origin/gh/yangw-dev/22/orig
2025-09-07T07:36:04.2022549Z  * [new branch]              gh/yangw-dev/23/base        -> origin/gh/yangw-dev/23/base
2025-09-07T07:36:04.2023714Z  * [new branch]              gh/yangw-dev/23/head        -> origin/gh/yangw-dev/23/head
2025-09-07T07:36:04.2024933Z  * [new branch]              gh/yangw-dev/23/orig        -> origin/gh/yangw-dev/23/orig
2025-09-07T07:36:04.2027057Z  * [new branch]              gh/yangw-dev/24/base        -> origin/gh/yangw-dev/24/base
2025-09-07T07:36:04.2028256Z  * [new branch]              gh/yangw-dev/24/head        -> origin/gh/yangw-dev/24/head
2025-09-07T07:36:04.2029417Z  * [new branch]              gh/yangw-dev/24/orig        -> origin/gh/yangw-dev/24/orig
2025-09-07T07:36:04.2031137Z  * [new branch]              gh/yangw-dev/25/base        -> origin/gh/yangw-dev/25/base
2025-09-07T07:36:04.2032298Z  * [new branch]              gh/yangw-dev/25/head        -> origin/gh/yangw-dev/25/head
2025-09-07T07:36:04.2033467Z  * [new branch]              gh/yangw-dev/25/orig        -> origin/gh/yangw-dev/25/orig
2025-09-07T07:36:04.2035177Z  * [new branch]              gh/yangw-dev/26/base        -> origin/gh/yangw-dev/26/base
2025-09-07T07:36:04.2036356Z  * [new branch]              gh/yangw-dev/26/head        -> origin/gh/yangw-dev/26/head
2025-09-07T07:36:04.2037526Z  * [new branch]              gh/yangw-dev/26/orig        -> origin/gh/yangw-dev/26/orig
2025-09-07T07:36:04.2039331Z  * [new branch]              gh/yangw-dev/27/base        -> origin/gh/yangw-dev/27/base
2025-09-07T07:36:04.2040546Z  * [new branch]              gh/yangw-dev/27/head        -> origin/gh/yangw-dev/27/head
2025-09-07T07:36:04.2041687Z  * [new branch]              gh/yangw-dev/27/orig        -> origin/gh/yangw-dev/27/orig
2025-09-07T07:36:04.2043865Z  * [new branch]              gh/ydwu4/233/base           -> origin/gh/ydwu4/233/base
2025-09-07T07:36:04.2045252Z  * [new branch]              gh/ydwu4/233/head           -> origin/gh/ydwu4/233/head
2025-09-07T07:36:04.2046445Z  * [new branch]              gh/ydwu4/233/orig           -> origin/gh/ydwu4/233/orig
2025-09-07T07:36:04.2048307Z  * [new branch]              gh/ydwu4/246/base           -> origin/gh/ydwu4/246/base
2025-09-07T07:36:04.2049898Z  * [new branch]              gh/ydwu4/246/head           -> origin/gh/ydwu4/246/head
2025-09-07T07:36:04.2051156Z  * [new branch]              gh/ydwu4/246/orig           -> origin/gh/ydwu4/246/orig
2025-09-07T07:36:04.2053028Z  * [new branch]              gh/ydwu4/253/base           -> origin/gh/ydwu4/253/base
2025-09-07T07:36:04.2054279Z  * [new branch]              gh/ydwu4/253/head           -> origin/gh/ydwu4/253/head
2025-09-07T07:36:04.2055495Z  * [new branch]              gh/ydwu4/253/orig           -> origin/gh/ydwu4/253/orig
2025-09-07T07:36:04.2057214Z  * [new branch]              gh/ydwu4/255/base           -> origin/gh/ydwu4/255/base
2025-09-07T07:36:04.2058573Z  * [new branch]              gh/ydwu4/255/head           -> origin/gh/ydwu4/255/head
2025-09-07T07:36:04.2059756Z  * [new branch]              gh/ydwu4/255/orig           -> origin/gh/ydwu4/255/orig
2025-09-07T07:36:04.2061900Z  * [new branch]              gh/ydwu4/259/base           -> origin/gh/ydwu4/259/base
2025-09-07T07:36:04.2063104Z  * [new branch]              gh/ydwu4/259/head           -> origin/gh/ydwu4/259/head
2025-09-07T07:36:04.2064292Z  * [new branch]              gh/ydwu4/259/orig           -> origin/gh/ydwu4/259/orig
2025-09-07T07:36:04.2066038Z  * [new branch]              gh/ydwu4/262/base           -> origin/gh/ydwu4/262/base
2025-09-07T07:36:04.2067265Z  * [new branch]              gh/ydwu4/262/head           -> origin/gh/ydwu4/262/head
2025-09-07T07:36:04.2068474Z  * [new branch]              gh/ydwu4/262/orig           -> origin/gh/ydwu4/262/orig
2025-09-07T07:36:04.2070188Z  * [new branch]              gh/ydwu4/263/base           -> origin/gh/ydwu4/263/base
2025-09-07T07:36:04.2071365Z  * [new branch]              gh/ydwu4/263/head           -> origin/gh/ydwu4/263/head
2025-09-07T07:36:04.2072571Z  * [new branch]              gh/ydwu4/263/orig           -> origin/gh/ydwu4/263/orig
2025-09-07T07:36:04.2074378Z  * [new branch]              gh/ydwu4/269/base           -> origin/gh/ydwu4/269/base
2025-09-07T07:36:04.2075547Z  * [new branch]              gh/ydwu4/269/head           -> origin/gh/ydwu4/269/head
2025-09-07T07:36:04.2076737Z  * [new branch]              gh/ydwu4/269/orig           -> origin/gh/ydwu4/269/orig
2025-09-07T07:36:04.2078481Z  * [new branch]              gh/ydwu4/270/base           -> origin/gh/ydwu4/270/base
2025-09-07T07:36:04.2079717Z  * [new branch]              gh/ydwu4/270/head           -> origin/gh/ydwu4/270/head
2025-09-07T07:36:04.2081487Z  * [new branch]              gh/ydwu4/270/orig           -> origin/gh/ydwu4/270/orig
2025-09-07T07:36:04.2083209Z  * [new branch]              gh/ydwu4/272/base           -> origin/gh/ydwu4/272/base
2025-09-07T07:36:04.2084488Z  * [new branch]              gh/ydwu4/272/head           -> origin/gh/ydwu4/272/head
2025-09-07T07:36:04.2085756Z  * [new branch]              gh/ydwu4/272/orig           -> origin/gh/ydwu4/272/orig
2025-09-07T07:36:04.2087322Z  * [new branch]              gh/ydwu4/275/base           -> origin/gh/ydwu4/275/base
2025-09-07T07:36:04.2088499Z  * [new branch]              gh/ydwu4/275/head           -> origin/gh/ydwu4/275/head
2025-09-07T07:36:04.2089663Z  * [new branch]              gh/ydwu4/275/orig           -> origin/gh/ydwu4/275/orig
2025-09-07T07:36:04.2091251Z  * [new branch]              gh/ydwu4/276/base           -> origin/gh/ydwu4/276/base
2025-09-07T07:36:04.2092440Z  * [new branch]              gh/ydwu4/276/head           -> origin/gh/ydwu4/276/head
2025-09-07T07:36:04.2093787Z  * [new branch]              gh/ydwu4/276/orig           -> origin/gh/ydwu4/276/orig
2025-09-07T07:36:04.2095786Z  * [new branch]              gh/ydwu4/279/base           -> origin/gh/ydwu4/279/base
2025-09-07T07:36:04.2097050Z  * [new branch]              gh/ydwu4/279/head           -> origin/gh/ydwu4/279/head
2025-09-07T07:36:04.2098379Z  * [new branch]              gh/ydwu4/279/orig           -> origin/gh/ydwu4/279/orig
2025-09-07T07:36:04.2100541Z  * [new branch]              gh/ydwu4/283/base           -> origin/gh/ydwu4/283/base
2025-09-07T07:36:04.2101814Z  * [new branch]              gh/ydwu4/283/head           -> origin/gh/ydwu4/283/head
2025-09-07T07:36:04.2102988Z  * [new branch]              gh/ydwu4/283/orig           -> origin/gh/ydwu4/283/orig
2025-09-07T07:36:04.2108824Z  * [new branch]              gh/ydwu4/289/base           -> origin/gh/ydwu4/289/base
2025-09-07T07:36:04.2110041Z  * [new branch]              gh/ydwu4/289/head           -> origin/gh/ydwu4/289/head
2025-09-07T07:36:04.2111219Z  * [new branch]              gh/ydwu4/289/orig           -> origin/gh/ydwu4/289/orig
2025-09-07T07:36:04.2113178Z  * [new branch]              gh/ydwu4/290/base           -> origin/gh/ydwu4/290/base
2025-09-07T07:36:04.2114290Z  * [new branch]              gh/ydwu4/290/head           -> origin/gh/ydwu4/290/head
2025-09-07T07:36:04.2115542Z  * [new branch]              gh/ydwu4/290/orig           -> origin/gh/ydwu4/290/orig
2025-09-07T07:36:04.2118108Z  * [new branch]              gh/ydwu4/291/base           -> origin/gh/ydwu4/291/base
2025-09-07T07:36:04.2119353Z  * [new branch]              gh/ydwu4/291/head           -> origin/gh/ydwu4/291/head
2025-09-07T07:36:04.2121069Z  * [new branch]              gh/ydwu4/291/orig           -> origin/gh/ydwu4/291/orig
2025-09-07T07:36:04.2123008Z  * [new branch]              gh/ydwu4/292/base           -> origin/gh/ydwu4/292/base
2025-09-07T07:36:04.2124258Z  * [new branch]              gh/ydwu4/292/head           -> origin/gh/ydwu4/292/head
2025-09-07T07:36:04.2125453Z  * [new branch]              gh/ydwu4/292/orig           -> origin/gh/ydwu4/292/orig
2025-09-07T07:36:04.2127192Z  * [new branch]              gh/ydwu4/293/base           -> origin/gh/ydwu4/293/base
2025-09-07T07:36:04.2128440Z  * [new branch]              gh/ydwu4/293/head           -> origin/gh/ydwu4/293/head
2025-09-07T07:36:04.2129645Z  * [new branch]              gh/ydwu4/293/orig           -> origin/gh/ydwu4/293/orig
2025-09-07T07:36:04.2131472Z  * [new branch]              gh/ydwu4/294/base           -> origin/gh/ydwu4/294/base
2025-09-07T07:36:04.2132633Z  * [new branch]              gh/ydwu4/294/head           -> origin/gh/ydwu4/294/head
2025-09-07T07:36:04.2133891Z  * [new branch]              gh/ydwu4/294/orig           -> origin/gh/ydwu4/294/orig
2025-09-07T07:36:04.2135662Z  * [new branch]              gh/ydwu4/295/base           -> origin/gh/ydwu4/295/base
2025-09-07T07:36:04.2136866Z  * [new branch]              gh/ydwu4/295/head           -> origin/gh/ydwu4/295/head
2025-09-07T07:36:04.2138121Z  * [new branch]              gh/ydwu4/295/orig           -> origin/gh/ydwu4/295/orig
2025-09-07T07:36:04.2139939Z  * [new branch]              gh/ydwu4/296/base           -> origin/gh/ydwu4/296/base
2025-09-07T07:36:04.2141083Z  * [new branch]              gh/ydwu4/296/head           -> origin/gh/ydwu4/296/head
2025-09-07T07:36:04.2142320Z  * [new branch]              gh/ydwu4/296/orig           -> origin/gh/ydwu4/296/orig
2025-09-07T07:36:04.2144727Z  * [new branch]              gh/ydwu4/300/base           -> origin/gh/ydwu4/300/base
2025-09-07T07:36:04.2146320Z  * [new branch]              gh/ydwu4/300/head           -> origin/gh/ydwu4/300/head
2025-09-07T07:36:04.2147615Z  * [new branch]              gh/ydwu4/300/orig           -> origin/gh/ydwu4/300/orig
2025-09-07T07:36:04.2150069Z  * [new branch]              gh/ydwu4/301/base           -> origin/gh/ydwu4/301/base
2025-09-07T07:36:04.2151225Z  * [new branch]              gh/ydwu4/301/head           -> origin/gh/ydwu4/301/head
2025-09-07T07:36:04.2152467Z  * [new branch]              gh/ydwu4/301/orig           -> origin/gh/ydwu4/301/orig
2025-09-07T07:36:04.2154189Z  * [new branch]              gh/ydwu4/302/base           -> origin/gh/ydwu4/302/base
2025-09-07T07:36:04.2155381Z  * [new branch]              gh/ydwu4/302/head           -> origin/gh/ydwu4/302/head
2025-09-07T07:36:04.2156580Z  * [new branch]              gh/ydwu4/302/orig           -> origin/gh/ydwu4/302/orig
2025-09-07T07:36:04.2158181Z  * [new branch]              gh/ydwu4/303/base           -> origin/gh/ydwu4/303/base
2025-09-07T07:36:04.2159474Z  * [new branch]              gh/ydwu4/303/head           -> origin/gh/ydwu4/303/head
2025-09-07T07:36:04.2160732Z  * [new branch]              gh/ydwu4/303/orig           -> origin/gh/ydwu4/303/orig
2025-09-07T07:36:04.2162789Z  * [new branch]              gh/ydwu4/304/base           -> origin/gh/ydwu4/304/base
2025-09-07T07:36:04.2164030Z  * [new branch]              gh/ydwu4/304/head           -> origin/gh/ydwu4/304/head
2025-09-07T07:36:04.2165237Z  * [new branch]              gh/ydwu4/304/orig           -> origin/gh/ydwu4/304/orig
2025-09-07T07:36:04.2167139Z  * [new branch]              gh/ydwu4/305/base           -> origin/gh/ydwu4/305/base
2025-09-07T07:36:04.2168411Z  * [new branch]              gh/ydwu4/305/head           -> origin/gh/ydwu4/305/head
2025-09-07T07:36:04.2169664Z  * [new branch]              gh/ydwu4/305/orig           -> origin/gh/ydwu4/305/orig
2025-09-07T07:36:04.2171576Z  * [new branch]              gh/ydwu4/306/base           -> origin/gh/ydwu4/306/base
2025-09-07T07:36:04.2172812Z  * [new branch]              gh/ydwu4/306/head           -> origin/gh/ydwu4/306/head
2025-09-07T07:36:04.2174016Z  * [new branch]              gh/ydwu4/306/orig           -> origin/gh/ydwu4/306/orig
2025-09-07T07:36:04.2175693Z  * [new branch]              gh/ydwu4/307/base           -> origin/gh/ydwu4/307/base
2025-09-07T07:36:04.2176831Z  * [new branch]              gh/ydwu4/307/head           -> origin/gh/ydwu4/307/head
2025-09-07T07:36:04.2178058Z  * [new branch]              gh/ydwu4/307/orig           -> origin/gh/ydwu4/307/orig
2025-09-07T07:36:04.2179858Z  * [new branch]              gh/ydwu4/308/base           -> origin/gh/ydwu4/308/base
2025-09-07T07:36:04.2181132Z  * [new branch]              gh/ydwu4/308/head           -> origin/gh/ydwu4/308/head
2025-09-07T07:36:04.2182285Z  * [new branch]              gh/ydwu4/308/orig           -> origin/gh/ydwu4/308/orig
2025-09-07T07:36:04.2183999Z  * [new branch]              gh/ydwu4/309/base           -> origin/gh/ydwu4/309/base
2025-09-07T07:36:04.2185144Z  * [new branch]              gh/ydwu4/309/head           -> origin/gh/ydwu4/309/head
2025-09-07T07:36:04.2186439Z  * [new branch]              gh/ydwu4/309/orig           -> origin/gh/ydwu4/309/orig
2025-09-07T07:36:04.2188286Z  * [new branch]              gh/ydwu4/310/base           -> origin/gh/ydwu4/310/base
2025-09-07T07:36:04.2189669Z  * [new branch]              gh/ydwu4/310/head           -> origin/gh/ydwu4/310/head
2025-09-07T07:36:04.2190842Z  * [new branch]              gh/ydwu4/310/orig           -> origin/gh/ydwu4/310/orig
2025-09-07T07:36:04.2192566Z  * [new branch]              gh/ydwu4/311/base           -> origin/gh/ydwu4/311/base
2025-09-07T07:36:04.2193738Z  * [new branch]              gh/ydwu4/311/head           -> origin/gh/ydwu4/311/head
2025-09-07T07:36:04.2195051Z  * [new branch]              gh/ydwu4/311/orig           -> origin/gh/ydwu4/311/orig
2025-09-07T07:36:04.2196782Z  * [new branch]              gh/ydwu4/312/base           -> origin/gh/ydwu4/312/base
2025-09-07T07:36:04.2198035Z  * [new branch]              gh/ydwu4/312/head           -> origin/gh/ydwu4/312/head
2025-09-07T07:36:04.2199304Z  * [new branch]              gh/ydwu4/312/orig           -> origin/gh/ydwu4/312/orig
2025-09-07T07:36:04.2201169Z  * [new branch]              gh/ydwu4/313/base           -> origin/gh/ydwu4/313/base
2025-09-07T07:36:04.2202509Z  * [new branch]              gh/ydwu4/313/head           -> origin/gh/ydwu4/313/head
2025-09-07T07:36:04.2204142Z  * [new branch]              gh/ydwu4/313/orig           -> origin/gh/ydwu4/313/orig
2025-09-07T07:36:04.2206184Z  * [new branch]              gh/ydwu4/314/base           -> origin/gh/ydwu4/314/base
2025-09-07T07:36:04.2207489Z  * [new branch]              gh/ydwu4/314/head           -> origin/gh/ydwu4/314/head
2025-09-07T07:36:04.2208702Z  * [new branch]              gh/ydwu4/314/orig           -> origin/gh/ydwu4/314/orig
2025-09-07T07:36:04.2210465Z  * [new branch]              gh/ydwu4/315/base           -> origin/gh/ydwu4/315/base
2025-09-07T07:36:04.2211747Z  * [new branch]              gh/ydwu4/315/head           -> origin/gh/ydwu4/315/head
2025-09-07T07:36:04.2212931Z  * [new branch]              gh/ydwu4/315/orig           -> origin/gh/ydwu4/315/orig
2025-09-07T07:36:04.2214782Z  * [new branch]              gh/ydwu4/316/base           -> origin/gh/ydwu4/316/base
2025-09-07T07:36:04.2215992Z  * [new branch]              gh/ydwu4/316/head           -> origin/gh/ydwu4/316/head
2025-09-07T07:36:04.2217231Z  * [new branch]              gh/ydwu4/316/orig           -> origin/gh/ydwu4/316/orig
2025-09-07T07:36:04.2219217Z  * [new branch]              gh/ydwu4/317/base           -> origin/gh/ydwu4/317/base
2025-09-07T07:36:04.2220214Z  * [new branch]              gh/ydwu4/317/head           -> origin/gh/ydwu4/317/head
2025-09-07T07:36:04.2221462Z  * [new branch]              gh/ydwu4/317/orig           -> origin/gh/ydwu4/317/orig
2025-09-07T07:36:04.2223237Z  * [new branch]              gh/ydwu4/318/base           -> origin/gh/ydwu4/318/base
2025-09-07T07:36:04.2224513Z  * [new branch]              gh/ydwu4/318/head           -> origin/gh/ydwu4/318/head
2025-09-07T07:36:04.2225686Z  * [new branch]              gh/ydwu4/318/orig           -> origin/gh/ydwu4/318/orig
2025-09-07T07:36:04.2227294Z  * [new branch]              gh/ydwu4/319/base           -> origin/gh/ydwu4/319/base
2025-09-07T07:36:04.2228470Z  * [new branch]              gh/ydwu4/319/head           -> origin/gh/ydwu4/319/head
2025-09-07T07:36:04.2229659Z  * [new branch]              gh/ydwu4/319/orig           -> origin/gh/ydwu4/319/orig
2025-09-07T07:36:04.2231663Z  * [new branch]              gh/ydwu4/320/base           -> origin/gh/ydwu4/320/base
2025-09-07T07:36:04.2232804Z  * [new branch]              gh/ydwu4/320/head           -> origin/gh/ydwu4/320/head
2025-09-07T07:36:04.2234025Z  * [new branch]              gh/ydwu4/320/orig           -> origin/gh/ydwu4/320/orig
2025-09-07T07:36:04.2235675Z  * [new branch]              gh/ydwu4/321/base           -> origin/gh/ydwu4/321/base
2025-09-07T07:36:04.2236946Z  * [new branch]              gh/ydwu4/321/head           -> origin/gh/ydwu4/321/head
2025-09-07T07:36:04.2238227Z  * [new branch]              gh/ydwu4/321/orig           -> origin/gh/ydwu4/321/orig
2025-09-07T07:36:04.2239976Z  * [new branch]              gh/ydwu4/322/base           -> origin/gh/ydwu4/322/base
2025-09-07T07:36:04.2241165Z  * [new branch]              gh/ydwu4/322/head           -> origin/gh/ydwu4/322/head
2025-09-07T07:36:04.2242354Z  * [new branch]              gh/ydwu4/322/orig           -> origin/gh/ydwu4/322/orig
2025-09-07T07:36:04.2244100Z  * [new branch]              gh/ydwu4/323/base           -> origin/gh/ydwu4/323/base
2025-09-07T07:36:04.2245270Z  * [new branch]              gh/ydwu4/323/head           -> origin/gh/ydwu4/323/head
2025-09-07T07:36:04.2246436Z  * [new branch]              gh/ydwu4/323/orig           -> origin/gh/ydwu4/323/orig
2025-09-07T07:36:04.2248216Z  * [new branch]              gh/ydwu4/324/base           -> origin/gh/ydwu4/324/base
2025-09-07T07:36:04.2249407Z  * [new branch]              gh/ydwu4/324/head           -> origin/gh/ydwu4/324/head
2025-09-07T07:36:04.2250586Z  * [new branch]              gh/ydwu4/324/orig           -> origin/gh/ydwu4/324/orig
2025-09-07T07:36:04.2252849Z  * [new branch]              gh/yf225/133/base           -> origin/gh/yf225/133/base
2025-09-07T07:36:04.2254069Z  * [new branch]              gh/yf225/133/head           -> origin/gh/yf225/133/head
2025-09-07T07:36:04.2256022Z  * [new branch]              gh/yf225/171/base           -> origin/gh/yf225/171/base
2025-09-07T07:36:04.2257300Z  * [new branch]              gh/yf225/171/head           -> origin/gh/yf225/171/head
2025-09-07T07:36:04.2258534Z  * [new branch]              gh/yf225/171/orig           -> origin/gh/yf225/171/orig
2025-09-07T07:36:04.2260323Z  * [new branch]              gh/yf225/172/base           -> origin/gh/yf225/172/base
2025-09-07T07:36:04.2261450Z  * [new branch]              gh/yf225/172/head           -> origin/gh/yf225/172/head
2025-09-07T07:36:04.2262594Z  * [new branch]              gh/yf225/172/orig           -> origin/gh/yf225/172/orig
2025-09-07T07:36:04.2264290Z  * [new branch]              gh/yf225/93/base            -> origin/gh/yf225/93/base
2025-09-07T07:36:04.2265478Z  * [new branch]              gh/yf225/93/head            -> origin/gh/yf225/93/head
2025-09-07T07:36:04.2268055Z  * [new branch]              gh/yifuwang/152/base        -> origin/gh/yifuwang/152/base
2025-09-07T07:36:04.2269608Z  * [new branch]              gh/yifuwang/152/head        -> origin/gh/yifuwang/152/head
2025-09-07T07:36:04.2271008Z  * [new branch]              gh/yifuwang/152/orig        -> origin/gh/yifuwang/152/orig
2025-09-07T07:36:04.2272587Z  * [new branch]              gh/yifuwang/195/base        -> origin/gh/yifuwang/195/base
2025-09-07T07:36:04.2273862Z  * [new branch]              gh/yifuwang/195/head        -> origin/gh/yifuwang/195/head
2025-09-07T07:36:04.2275072Z  * [new branch]              gh/yifuwang/195/orig        -> origin/gh/yifuwang/195/orig
2025-09-07T07:36:04.2277218Z  * [new branch]              gh/yiming0416/1/base        -> origin/gh/yiming0416/1/base
2025-09-07T07:36:04.2278411Z  * [new branch]              gh/yiming0416/1/head        -> origin/gh/yiming0416/1/head
2025-09-07T07:36:04.2279984Z  * [new branch]              gh/yiming0416/2/base        -> origin/gh/yiming0416/2/base
2025-09-07T07:36:04.2281113Z  * [new branch]              gh/yiming0416/2/head        -> origin/gh/yiming0416/2/head
2025-09-07T07:36:04.2283243Z  * [new branch]              gh/ysiraichi/79/base        -> origin/gh/ysiraichi/79/base
2025-09-07T07:36:04.2284453Z  * [new branch]              gh/ysiraichi/79/head        -> origin/gh/ysiraichi/79/head
2025-09-07T07:36:04.2285889Z  * [new branch]              gh/ysiraichi/79/orig        -> origin/gh/ysiraichi/79/orig
2025-09-07T07:36:04.2287557Z  * [new branch]              gh/ysiraichi/88/base        -> origin/gh/ysiraichi/88/base
2025-09-07T07:36:04.2288737Z  * [new branch]              gh/ysiraichi/88/head        -> origin/gh/ysiraichi/88/head
2025-09-07T07:36:04.2289931Z  * [new branch]              gh/ysiraichi/88/orig        -> origin/gh/ysiraichi/88/orig
2025-09-07T07:36:04.2292222Z  * [new branch]              gh/zhxchen17/25/base        -> origin/gh/zhxchen17/25/base
2025-09-07T07:36:04.2293418Z  * [new branch]              gh/zhxchen17/25/head        -> origin/gh/zhxchen17/25/head
2025-09-07T07:36:04.2294614Z  * [new branch]              gh/zhxchen17/25/orig        -> origin/gh/zhxchen17/25/orig
2025-09-07T07:36:04.2296419Z  * [new branch]              gh/zhxchen17/31/base        -> origin/gh/zhxchen17/31/base
2025-09-07T07:36:04.2297696Z  * [new branch]              gh/zhxchen17/31/head        -> origin/gh/zhxchen17/31/head
2025-09-07T07:36:04.2298957Z  * [new branch]              gh/zhxchen17/31/orig        -> origin/gh/zhxchen17/31/orig
2025-09-07T07:36:04.2300690Z  * [new branch]              gh/zhxchen17/34/base        -> origin/gh/zhxchen17/34/base
2025-09-07T07:36:04.2301972Z  * [new branch]              gh/zhxchen17/34/head        -> origin/gh/zhxchen17/34/head
2025-09-07T07:36:04.2303723Z  * [new branch]              gh/zhxchen17/35/base        -> origin/gh/zhxchen17/35/base
2025-09-07T07:36:04.2304998Z  * [new branch]              gh/zhxchen17/35/head        -> origin/gh/zhxchen17/35/head
2025-09-07T07:36:04.2307087Z  * [new branch]              gh/zhxchen17/37/base        -> origin/gh/zhxchen17/37/base
2025-09-07T07:36:04.2308394Z  * [new branch]              gh/zhxchen17/37/head        -> origin/gh/zhxchen17/37/head
2025-09-07T07:36:04.2309665Z  * [new branch]              gh/zhxchen17/37/orig        -> origin/gh/zhxchen17/37/orig
2025-09-07T07:36:04.2311622Z  * [new branch]              gh/zhxchen17/38/base        -> origin/gh/zhxchen17/38/base
2025-09-07T07:36:04.2312754Z  * [new branch]              gh/zhxchen17/38/head        -> origin/gh/zhxchen17/38/head
2025-09-07T07:36:04.2314011Z  * [new branch]              gh/zhxchen17/38/orig        -> origin/gh/zhxchen17/38/orig
2025-09-07T07:36:04.2316078Z  * [new branch]              gh/zhxchen17/39/base        -> origin/gh/zhxchen17/39/base
2025-09-07T07:36:04.2317435Z  * [new branch]              gh/zhxchen17/39/head        -> origin/gh/zhxchen17/39/head
2025-09-07T07:36:04.2318718Z  * [new branch]              gh/zhxchen17/39/orig        -> origin/gh/zhxchen17/39/orig
2025-09-07T07:36:04.2334778Z  * [new branch]              gh/zhxchen17/40/base        -> origin/gh/zhxchen17/40/base
2025-09-07T07:36:04.2335360Z  * [new branch]              gh/zhxchen17/40/head        -> origin/gh/zhxchen17/40/head
2025-09-07T07:36:04.2336012Z  * [new branch]              gh/zhxchen17/40/orig        -> origin/gh/zhxchen17/40/orig
2025-09-07T07:36:04.2336160Z  * [new branch]              gh/zhxchen17/41/base        -> origin/gh/zhxchen17/41/base
2025-09-07T07:36:04.2336294Z  * [new branch]              gh/zhxchen17/41/head        -> origin/gh/zhxchen17/41/head
2025-09-07T07:36:04.2336431Z  * [new branch]              gh/zhxchen17/41/orig        -> origin/gh/zhxchen17/41/orig
2025-09-07T07:36:04.2336559Z  * [new branch]              gh/zhxchen17/42/base        -> origin/gh/zhxchen17/42/base
2025-09-07T07:36:04.2336688Z  * [new branch]              gh/zhxchen17/42/head        -> origin/gh/zhxchen17/42/head
2025-09-07T07:36:04.2336819Z  * [new branch]              gh/zhxchen17/42/orig        -> origin/gh/zhxchen17/42/orig
2025-09-07T07:36:04.2336952Z  * [new branch]              gh/zhxchen17/43/base        -> origin/gh/zhxchen17/43/base
2025-09-07T07:36:04.2337122Z  * [new branch]              gh/zhxchen17/43/head        -> origin/gh/zhxchen17/43/head
2025-09-07T07:36:04.2337253Z  * [new branch]              gh/zhxchen17/43/orig        -> origin/gh/zhxchen17/43/orig
2025-09-07T07:36:04.2339134Z  * [new branch]              gh/zhxchen17/44/base        -> origin/gh/zhxchen17/44/base
2025-09-07T07:36:04.2340264Z  * [new branch]              gh/zhxchen17/44/head        -> origin/gh/zhxchen17/44/head
2025-09-07T07:36:04.2341454Z  * [new branch]              gh/zhxchen17/44/orig        -> origin/gh/zhxchen17/44/orig
2025-09-07T07:36:04.2343144Z  * [new branch]              gh/zhxchen17/45/base        -> origin/gh/zhxchen17/45/base
2025-09-07T07:36:04.2344639Z  * [new branch]              gh/zhxchen17/45/head        -> origin/gh/zhxchen17/45/head
2025-09-07T07:36:04.2345903Z  * [new branch]              gh/zhxchen17/45/orig        -> origin/gh/zhxchen17/45/orig
2025-09-07T07:36:04.2348042Z  * [new branch]              gh/zklaus/10/base           -> origin/gh/zklaus/10/base
2025-09-07T07:36:04.2349271Z  * [new branch]              gh/zklaus/10/head           -> origin/gh/zklaus/10/head
2025-09-07T07:36:04.2350467Z  * [new branch]              gh/zklaus/10/orig           -> origin/gh/zklaus/10/orig
2025-09-07T07:36:04.2352201Z  * [new branch]              gh/zklaus/11/base           -> origin/gh/zklaus/11/base
2025-09-07T07:36:04.2353370Z  * [new branch]              gh/zklaus/11/head           -> origin/gh/zklaus/11/head
2025-09-07T07:36:04.2354691Z  * [new branch]              gh/zklaus/11/orig           -> origin/gh/zklaus/11/orig
2025-09-07T07:36:04.2356332Z  * [new branch]              gh/zklaus/12/base           -> origin/gh/zklaus/12/base
2025-09-07T07:36:04.2357543Z  * [new branch]              gh/zklaus/12/head           -> origin/gh/zklaus/12/head
2025-09-07T07:36:04.2358704Z  * [new branch]              gh/zklaus/12/orig           -> origin/gh/zklaus/12/orig
2025-09-07T07:36:04.2360367Z  * [new branch]              gh/zklaus/14/base           -> origin/gh/zklaus/14/base
2025-09-07T07:36:04.2361525Z  * [new branch]              gh/zklaus/14/head           -> origin/gh/zklaus/14/head
2025-09-07T07:36:04.2362733Z  * [new branch]              gh/zklaus/14/orig           -> origin/gh/zklaus/14/orig
2025-09-07T07:36:04.2364408Z  * [new branch]              gh/zklaus/15/base           -> origin/gh/zklaus/15/base
2025-09-07T07:36:04.2365666Z  * [new branch]              gh/zklaus/15/head           -> origin/gh/zklaus/15/head
2025-09-07T07:36:04.2366825Z  * [new branch]              gh/zklaus/15/orig           -> origin/gh/zklaus/15/orig
2025-09-07T07:36:04.2368538Z  * [new branch]              gh/zklaus/16/base           -> origin/gh/zklaus/16/base
2025-09-07T07:36:04.2369773Z  * [new branch]              gh/zklaus/16/head           -> origin/gh/zklaus/16/head
2025-09-07T07:36:04.2371001Z  * [new branch]              gh/zklaus/16/orig           -> origin/gh/zklaus/16/orig
2025-09-07T07:36:04.2372746Z  * [new branch]              gh/zklaus/17/base           -> origin/gh/zklaus/17/base
2025-09-07T07:36:04.2373912Z  * [new branch]              gh/zklaus/17/head           -> origin/gh/zklaus/17/head
2025-09-07T07:36:04.2375198Z  * [new branch]              gh/zklaus/17/orig           -> origin/gh/zklaus/17/orig
2025-09-07T07:36:04.2376747Z  * [new branch]              gh/zklaus/18/base           -> origin/gh/zklaus/18/base
2025-09-07T07:36:04.2378060Z  * [new branch]              gh/zklaus/18/head           -> origin/gh/zklaus/18/head
2025-09-07T07:36:04.2379410Z  * [new branch]              gh/zklaus/18/orig           -> origin/gh/zklaus/18/orig
2025-09-07T07:36:04.2381067Z  * [new branch]              gh/zklaus/19/base           -> origin/gh/zklaus/19/base
2025-09-07T07:36:04.2382313Z  * [new branch]              gh/zklaus/19/head           -> origin/gh/zklaus/19/head
2025-09-07T07:36:04.2383469Z  * [new branch]              gh/zklaus/19/orig           -> origin/gh/zklaus/19/orig
2025-09-07T07:36:04.2385140Z  * [new branch]              gh/zklaus/20/base           -> origin/gh/zklaus/20/base
2025-09-07T07:36:04.2386296Z  * [new branch]              gh/zklaus/20/head           -> origin/gh/zklaus/20/head
2025-09-07T07:36:04.2387593Z  * [new branch]              gh/zklaus/20/orig           -> origin/gh/zklaus/20/orig
2025-09-07T07:36:04.2389762Z  * [new branch]              gh/zklaus/7/base            -> origin/gh/zklaus/7/base
2025-09-07T07:36:04.2390950Z  * [new branch]              gh/zklaus/7/head            -> origin/gh/zklaus/7/head
2025-09-07T07:36:04.2392134Z  * [new branch]              gh/zklaus/7/orig            -> origin/gh/zklaus/7/orig
2025-09-07T07:36:04.2393771Z  * [new branch]              gh/zklaus/9/base            -> origin/gh/zklaus/9/base
2025-09-07T07:36:04.2394970Z  * [new branch]              gh/zklaus/9/head            -> origin/gh/zklaus/9/head
2025-09-07T07:36:04.2396139Z  * [new branch]              gh/zklaus/9/orig            -> origin/gh/zklaus/9/orig
2025-09-07T07:36:04.2398328Z  * [new branch]              gh/zou3519/1175/base        -> origin/gh/zou3519/1175/base
2025-09-07T07:36:04.2399505Z  * [new branch]              gh/zou3519/1175/head        -> origin/gh/zou3519/1175/head
2025-09-07T07:36:04.2400701Z  * [new branch]              gh/zou3519/1175/orig        -> origin/gh/zou3519/1175/orig
2025-09-07T07:36:04.2402436Z  * [new branch]              gh/zou3519/1177/base        -> origin/gh/zou3519/1177/base
2025-09-07T07:36:04.2403954Z  * [new branch]              gh/zou3519/1177/head        -> origin/gh/zou3519/1177/head
2025-09-07T07:36:04.2405140Z  * [new branch]              gh/zou3519/1177/orig        -> origin/gh/zou3519/1177/orig
2025-09-07T07:36:04.2406896Z  * [new branch]              gh/zou3519/1191/base        -> origin/gh/zou3519/1191/base
2025-09-07T07:36:04.2408217Z  * [new branch]              gh/zou3519/1191/head        -> origin/gh/zou3519/1191/head
2025-09-07T07:36:04.2409432Z  * [new branch]              gh/zou3519/1191/orig        -> origin/gh/zou3519/1191/orig
2025-09-07T07:36:04.2411272Z  * [new branch]              gh/zou3519/1192/base        -> origin/gh/zou3519/1192/base
2025-09-07T07:36:04.2412486Z  * [new branch]              gh/zou3519/1192/head        -> origin/gh/zou3519/1192/head
2025-09-07T07:36:04.2413687Z  * [new branch]              gh/zou3519/1192/orig        -> origin/gh/zou3519/1192/orig
2025-09-07T07:36:04.2415388Z  * [new branch]              gh/zou3519/1193/base        -> origin/gh/zou3519/1193/base
2025-09-07T07:36:04.2416586Z  * [new branch]              gh/zou3519/1193/head        -> origin/gh/zou3519/1193/head
2025-09-07T07:36:04.2417836Z  * [new branch]              gh/zou3519/1193/orig        -> origin/gh/zou3519/1193/orig
2025-09-07T07:36:04.2419513Z  * [new branch]              gh/zou3519/1194/base        -> origin/gh/zou3519/1194/base
2025-09-07T07:36:04.2420838Z  * [new branch]              gh/zou3519/1194/head        -> origin/gh/zou3519/1194/head
2025-09-07T07:36:04.2422074Z  * [new branch]              gh/zou3519/1194/orig        -> origin/gh/zou3519/1194/orig
2025-09-07T07:36:04.2423844Z  * [new branch]              gh/zou3519/1195/base        -> origin/gh/zou3519/1195/base
2025-09-07T07:36:04.2425559Z  * [new branch]              gh/zou3519/1195/head        -> origin/gh/zou3519/1195/head
2025-09-07T07:36:04.2426413Z  * [new branch]              gh/zou3519/1195/orig        -> origin/gh/zou3519/1195/orig
2025-09-07T07:36:04.2427983Z  * [new branch]              gh/zou3519/1196/base        -> origin/gh/zou3519/1196/base
2025-09-07T07:36:04.2429221Z  * [new branch]              gh/zou3519/1196/head        -> origin/gh/zou3519/1196/head
2025-09-07T07:36:04.2430411Z  * [new branch]              gh/zou3519/1196/orig        -> origin/gh/zou3519/1196/orig
2025-09-07T07:36:04.2431981Z  * [new branch]              gh/zou3519/1197/base        -> origin/gh/zou3519/1197/base
2025-09-07T07:36:04.2433179Z  * [new branch]              gh/zou3519/1197/head        -> origin/gh/zou3519/1197/head
2025-09-07T07:36:04.2434513Z  * [new branch]              gh/zou3519/1197/orig        -> origin/gh/zou3519/1197/orig
2025-09-07T07:36:04.2436743Z  * [new branch]              gh/zpcore/1/base            -> origin/gh/zpcore/1/base
2025-09-07T07:36:04.2437925Z  * [new branch]              gh/zpcore/1/head            -> origin/gh/zpcore/1/head
2025-09-07T07:36:04.2439681Z  * [new branch]              gh/zpcore/10/base           -> origin/gh/zpcore/10/base
2025-09-07T07:36:04.2440730Z  * [new branch]              gh/zpcore/10/head           -> origin/gh/zpcore/10/head
2025-09-07T07:36:04.2441906Z  * [new branch]              gh/zpcore/10/orig           -> origin/gh/zpcore/10/orig
2025-09-07T07:36:04.2444143Z  * [new branch]              gh/zpcore/11/base           -> origin/gh/zpcore/11/base
2025-09-07T07:36:04.2445394Z  * [new branch]              gh/zpcore/11/head           -> origin/gh/zpcore/11/head
2025-09-07T07:36:04.2446598Z  * [new branch]              gh/zpcore/11/orig           -> origin/gh/zpcore/11/orig
2025-09-07T07:36:04.2448532Z  * [new branch]              gh/zpcore/12/base           -> origin/gh/zpcore/12/base
2025-09-07T07:36:04.2449955Z  * [new branch]              gh/zpcore/12/head           -> origin/gh/zpcore/12/head
2025-09-07T07:36:04.2451541Z  * [new branch]              gh/zpcore/12/orig           -> origin/gh/zpcore/12/orig
2025-09-07T07:36:04.2453309Z  * [new branch]              gh/zpcore/13/base           -> origin/gh/zpcore/13/base
2025-09-07T07:36:04.2454650Z  * [new branch]              gh/zpcore/13/head           -> origin/gh/zpcore/13/head
2025-09-07T07:36:04.2455793Z  * [new branch]              gh/zpcore/13/orig           -> origin/gh/zpcore/13/orig
2025-09-07T07:36:04.2457520Z  * [new branch]              gh/zpcore/14/base           -> origin/gh/zpcore/14/base
2025-09-07T07:36:04.2458789Z  * [new branch]              gh/zpcore/14/head           -> origin/gh/zpcore/14/head
2025-09-07T07:36:04.2460599Z  * [new branch]              gh/zpcore/2/base            -> origin/gh/zpcore/2/base
2025-09-07T07:36:04.2461828Z  * [new branch]              gh/zpcore/2/head            -> origin/gh/zpcore/2/head
2025-09-07T07:36:04.2463402Z  * [new branch]              gh/zpcore/3/base            -> origin/gh/zpcore/3/base
2025-09-07T07:36:04.2464572Z  * [new branch]              gh/zpcore/3/head            -> origin/gh/zpcore/3/head
2025-09-07T07:36:04.2466137Z  * [new branch]              gh/zpcore/4/base            -> origin/gh/zpcore/4/base
2025-09-07T07:36:04.2467269Z  * [new branch]              gh/zpcore/4/head            -> origin/gh/zpcore/4/head
2025-09-07T07:36:04.2469285Z  * [new branch]              gh/zpcore/5/base            -> origin/gh/zpcore/5/base
2025-09-07T07:36:04.2470536Z  * [new branch]              gh/zpcore/5/head            -> origin/gh/zpcore/5/head
2025-09-07T07:36:04.2472123Z  * [new branch]              gh/zpcore/6/base            -> origin/gh/zpcore/6/base
2025-09-07T07:36:04.2473329Z  * [new branch]              gh/zpcore/6/head            -> origin/gh/zpcore/6/head
2025-09-07T07:36:04.2474914Z  * [new branch]              gh/zpcore/7/base            -> origin/gh/zpcore/7/base
2025-09-07T07:36:04.2476021Z  * [new branch]              gh/zpcore/7/head            -> origin/gh/zpcore/7/head
2025-09-07T07:36:04.2477710Z  * [new branch]              gh/zpcore/8/base            -> origin/gh/zpcore/8/base
2025-09-07T07:36:04.2478813Z  * [new branch]              gh/zpcore/8/head            -> origin/gh/zpcore/8/head
2025-09-07T07:36:04.2480226Z  * [new branch]              google-main                 -> origin/google-main
2025-09-07T07:36:04.2482068Z  * [new branch]              guangyey/external_stream    -> origin/guangyey/external_stream
2025-09-07T07:36:04.2483661Z  * [new branch]              guangyey/host_alloc         -> origin/guangyey/host_alloc
2025-09-07T07:36:04.2484680Z  * [new branch]              guangyey/reimport           -> origin/guangyey/reimport
2025-09-07T07:36:04.2485906Z  * [new branch]              guangyey/test_2025          -> origin/guangyey/test_2025
2025-09-07T07:36:04.2487913Z  * [new branch]              guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9
2025-09-07T07:36:04.2489760Z  * [new branch]              haozhe/bf16-dynamic-shape   -> origin/haozhe/bf16-dynamic-shape
2025-09-07T07:36:04.2491020Z  * [new branch]              hc_baseline                 -> origin/hc_baseline
2025-09-07T07:36:04.2492451Z  * [new branch]              hf_update                   -> origin/hf_update
2025-09-07T07:36:04.2493681Z  * [new branch]              hhh_decomp_mul              -> origin/hhh_decomp_mul
2025-09-07T07:36:04.2494944Z  * [new branch]              hhh_rand                    -> origin/hhh_rand
2025-09-07T07:36:04.2496682Z  * [new branch]              hoy/mmsplitk                -> origin/hoy/mmsplitk
2025-09-07T07:36:04.2497817Z  * [new branch]              hoy/triton-PR3973           -> origin/hoy/triton-PR3973
2025-09-07T07:36:04.2499140Z  * [new branch]              hoy/triton-coalescing-baseline -> origin/hoy/triton-coalescing-baseline
2025-09-07T07:36:04.2500240Z  * [new branch]              hoy/triton-coalescing-new   -> origin/hoy/triton-coalescing-new
2025-09-07T07:36:04.2501356Z  * [new branch]              hoy/triton-coalescing-vec   -> origin/hoy/triton-coalescing-vec
2025-09-07T07:36:04.2502610Z  * [new branch]              inductordecompfix           -> origin/inductordecompfix
2025-09-07T07:36:04.2506135Z  * [new branch]              inline                      -> origin/inline
2025-09-07T07:36:04.2507512Z  * [new branch]              inlining                    -> origin/inlining
2025-09-07T07:36:04.2508780Z  * [new branch]              inlining-ezyang             -> origin/inlining-ezyang
2025-09-07T07:36:04.2510073Z  * [new branch]              install-torchao-0.13.0      -> origin/install-torchao-0.13.0
2025-09-07T07:36:04.2511296Z  * [new branch]              int8_sdpa                   -> origin/int8_sdpa
2025-09-07T07:36:04.2512584Z  * [new branch]              invoke-subgraph             -> origin/invoke-subgraph
2025-09-07T07:36:04.2513886Z  * [new branch]              issue#58739                 -> origin/issue#58739
2025-09-07T07:36:04.2515915Z  * [new branch]              jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2
2025-09-07T07:36:04.2516935Z  * [new branch]              jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2
2025-09-07T07:36:04.2518658Z  * [new branch]              jeanschmidt/disable_rocm_build_tests -> origin/jeanschmidt/disable_rocm_build_tests
2025-09-07T07:36:04.2519967Z  * [new branch]              jithunnair-amd-patch-1      -> origin/jithunnair-amd-patch-1
2025-09-07T07:36:04.2521237Z  * [new branch]              jithunnair-amd-patch-2      -> origin/jithunnair-amd-patch-2
2025-09-07T07:36:04.2522913Z  * [new branch]              justinchu/attention-tests   -> origin/justinchu/attention-tests
2025-09-07T07:36:04.2524058Z  * [new branch]              justinchu/native-qdq        -> origin/justinchu/native-qdq
2025-09-07T07:36:04.2525447Z  * [new branch]              justinchu/ort-122           -> origin/justinchu/ort-122
2025-09-07T07:36:04.2527190Z  * [new branch]              justinchuby/dynamo-true     -> origin/justinchuby/dynamo-true
2025-09-07T07:36:04.2529022Z  * [new branch]              kainan666/xlf_debug         -> origin/kainan666/xlf_debug
2025-09-07T07:36:04.2530154Z  * [new branch]              kainan_test                 -> origin/kainan_test
2025-09-07T07:36:04.2531419Z  * [new branch]              learnablebias               -> origin/learnablebias
2025-09-07T07:36:04.2533132Z  * [new branch]              leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues
2025-09-07T07:36:04.2534904Z  * [new branch]              lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error
2025-09-07T07:36:04.2536540Z  * [new branch]              liaoxuan/shm_all_reduce     -> origin/liaoxuan/shm_all_reduce
2025-09-07T07:36:04.2537737Z  * [new branch]              liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax
2025-09-07T07:36:04.2538867Z  * [new branch]              liaoxuan/test_int8_sdpa     -> origin/liaoxuan/test_int8_sdpa
2025-09-07T07:36:04.2540137Z  * [new branch]              lintbuilddocker             -> origin/lintbuilddocker
2025-09-07T07:36:04.2541316Z  * [new branch]              llama4-stable               -> origin/llama4-stable
2025-09-07T07:36:04.2542632Z  * [new branch]              logdetfix                   -> origin/logdetfix
2025-09-07T07:36:04.2544886Z  * [new branch]              lts/release/1.8             -> origin/lts/release/1.8
2025-09-07T07:36:04.2546648Z  * [new branch]              lucaskabela/#94773          -> origin/lucaskabela/#94773
2025-09-07T07:36:04.2547816Z  * [new branch]              lucaskabela/flop_counter    -> origin/lucaskabela/flop_counter
2025-09-07T07:36:04.2548939Z  * [new branch]              lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp
2025-09-07T07:36:04.2550067Z  * [new branch]              lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo
2025-09-07T07:36:04.2551251Z  * [new branch]              lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr
2025-09-07T07:36:04.2552319Z  * [new branch]              lucaskabela/issue_120648    -> origin/lucaskabela/issue_120648
2025-09-07T07:36:04.2553798Z  * [new branch]              lucaskabela/misc_typing_dynamo -> origin/lucaskabela/misc_typing_dynamo
2025-09-07T07:36:04.2555496Z  * [new branch]              lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr
2025-09-07T07:36:04.2556761Z  * [new branch]              lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata
2025-09-07T07:36:04.2557927Z  * [new branch]              lucaskabela/rnn_decomp      -> origin/lucaskabela/rnn_decomp
2025-09-07T07:36:04.2559247Z  * [new branch]              lucaskabela/typing_backends -> origin/lucaskabela/typing_backends
2025-09-07T07:36:04.2560449Z  * [new branch]              lucaskabela/typing_symbolic_convert -> origin/lucaskabela/typing_symbolic_convert
2025-09-07T07:36:04.2561732Z  * [new branch]              lucaskabela/typing_utils_improvements -> origin/lucaskabela/typing_utils_improvements
2025-09-07T07:36:04.2563403Z  * [new branch]              main                        -> origin/main
2025-09-07T07:36:04.2564910Z  * [new branch]              main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests
2025-09-07T07:36:04.2566115Z  * [new branch]              malfet-patch-1              -> origin/malfet-patch-1
2025-09-07T07:36:04.2567461Z  * [new branch]              malfet-patch-12             -> origin/malfet-patch-12
2025-09-07T07:36:04.2568784Z  * [new branch]              malfet-patch-14             -> origin/malfet-patch-14
2025-09-07T07:36:04.2570133Z  * [new branch]              malfet-patch-6              -> origin/malfet-patch-6
2025-09-07T07:36:04.2571446Z  * [new branch]              malfet-patch-8              -> origin/malfet-patch-8
2025-09-07T07:36:04.2573419Z  * [new branch]              malfet/be-move-more-settings-to-checkout-pytorch -> origin/malfet/be-move-more-settings-to-checkout-pytorch
2025-09-07T07:36:04.2574615Z  * [new branch]              malfet/delete-upsteam-cuda  -> origin/malfet/delete-upsteam-cuda
2025-09-07T07:36:04.2575699Z  * [new branch]              malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im
2025-09-07T07:36:04.2577386Z  * [new branch]              manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps
2025-09-07T07:36:04.2578835Z  * [new branch]              metascroy-patch-1           -> origin/metascroy-patch-1
2025-09-07T07:36:04.2580577Z  * [new branch]              mlazos/S429861-debug        -> origin/mlazos/S429861-debug
2025-09-07T07:36:04.2581675Z  * [new branch]              mlazos/aa                   -> origin/mlazos/aa
2025-09-07T07:36:04.2582823Z  * [new branch]              mlazos/arg-renames          -> origin/mlazos/arg-renames
2025-09-07T07:36:04.2584021Z  * [new branch]              mlazos/backup-test-branch   -> origin/mlazos/backup-test-branch
2025-09-07T07:36:04.2585144Z  * [new branch]              mlazos/bad-cudagraphs       -> origin/mlazos/bad-cudagraphs
2025-09-07T07:36:04.2586289Z  * [new branch]              mlazos/baseline             -> origin/mlazos/baseline
2025-09-07T07:36:04.2587433Z  * [new branch]              mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks
2025-09-07T07:36:04.2588550Z  * [new branch]              mlazos/beta-tensor          -> origin/mlazos/beta-tensor
2025-09-07T07:36:04.2589952Z  * [new branch]              mlazos/better-msg           -> origin/mlazos/better-msg
2025-09-07T07:36:04.2592098Z  * [new branch]              mlazos/buffers              -> origin/mlazos/buffers
2025-09-07T07:36:04.2593276Z  * [new branch]              mlazos/buffers2             -> origin/mlazos/buffers2
2025-09-07T07:36:04.2594808Z  * [new branch]              mlazos/buffers3             -> origin/mlazos/buffers3
2025-09-07T07:36:04.2596350Z  * [new branch]              mlazos/ck2                  -> origin/mlazos/ck2
2025-09-07T07:36:04.2597788Z  * [new branch]              mlazos/combokernels         -> origin/mlazos/combokernels
2025-09-07T07:36:04.2599061Z  * [new branch]              mlazos/ctx-cleanup          -> origin/mlazos/ctx-cleanup
2025-09-07T07:36:04.2600178Z  * [new branch]              mlazos/cuda-cmd-log         -> origin/mlazos/cuda-cmd-log
2025-09-07T07:36:04.2601518Z  * [new branch]              mlazos/cudagraph-tests      -> origin/mlazos/cudagraph-tests
2025-09-07T07:36:04.2602755Z  * [new branch]              mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement
2025-09-07T07:36:04.2604206Z  * [new branch]              mlazos/cutlass-test         -> origin/mlazos/cutlass-test
2025-09-07T07:36:04.2605484Z  * [new branch]              mlazos/cutlass-topo-bug     -> origin/mlazos/cutlass-topo-bug
2025-09-07T07:36:04.2606654Z  * [new branch]              mlazos/data-gather          -> origin/mlazos/data-gather
2025-09-07T07:36:04.2607902Z  * [new branch]              mlazos/data-ptrs2           -> origin/mlazos/data-ptrs2
2025-09-07T07:36:04.2609048Z  * [new branch]              mlazos/data-ptrs3           -> origin/mlazos/data-ptrs3
2025-09-07T07:36:04.2610284Z  * [new branch]              mlazos/dataclass-proxy      -> origin/mlazos/dataclass-proxy
2025-09-07T07:36:04.2611464Z  * [new branch]              mlazos/dc-attrs             -> origin/mlazos/dc-attrs
2025-09-07T07:36:04.2612765Z  * [new branch]              mlazos/dc-helion            -> origin/mlazos/dc-helion
2025-09-07T07:36:04.2613890Z  * [new branch]              mlazos/dict-fix             -> origin/mlazos/dict-fix
2025-09-07T07:36:04.2615168Z  * [new branch]              mlazos/disable-closures     -> origin/mlazos/disable-closures
2025-09-07T07:36:04.2616402Z  * [new branch]              mlazos/disable-tf           -> origin/mlazos/disable-tf
2025-09-07T07:36:04.2617447Z  * [new branch]              mlazos/dupe-fix             -> origin/mlazos/dupe-fix
2025-09-07T07:36:04.2618918Z  * [new branch]              mlazos/dyn-batch            -> origin/mlazos/dyn-batch
2025-09-07T07:36:04.2620236Z  * [new branch]              mlazos/evt                  -> origin/mlazos/evt
2025-09-07T07:36:04.2621452Z  * [new branch]              mlazos/exp_disable          -> origin/mlazos/exp_disable
2025-09-07T07:36:04.2622661Z  * [new branch]              mlazos/extract-examples     -> origin/mlazos/extract-examples
2025-09-07T07:36:04.2623827Z  * [new branch]              mlazos/foreach-op           -> origin/mlazos/foreach-op
2025-09-07T07:36:04.2625017Z  * [new branch]              mlazos/fp8                  -> origin/mlazos/fp8
2025-09-07T07:36:04.2626404Z  * [new branch]              mlazos/fp8-bias             -> origin/mlazos/fp8-bias
2025-09-07T07:36:04.2627632Z  * [new branch]              mlazos/fp8-bias-fusion      -> origin/mlazos/fp8-bias-fusion
2025-09-07T07:36:04.2628680Z  * [new branch]              mlazos/fp8-fixes            -> origin/mlazos/fp8-fixes
2025-09-07T07:36:04.2629905Z  * [new branch]              mlazos/freezing             -> origin/mlazos/freezing
2025-09-07T07:36:04.2631143Z  * [new branch]              mlazos/h-comp               -> origin/mlazos/h-comp
2025-09-07T07:36:04.2632427Z  * [new branch]              mlazos/h-comp2              -> origin/mlazos/h-comp2
2025-09-07T07:36:04.2633661Z  * [new branch]              mlazos/hash-hop             -> origin/mlazos/hash-hop
2025-09-07T07:36:04.2634897Z  * [new branch]              mlazos/hc                   -> origin/mlazos/hc
2025-09-07T07:36:04.2636183Z  * [new branch]              mlazos/hc-cycles            -> origin/mlazos/hc-cycles
2025-09-07T07:36:04.2637800Z  * [new branch]              mlazos/hc-fixes             -> origin/mlazos/hc-fixes
2025-09-07T07:36:04.2639091Z  * [new branch]              mlazos/hc-fixes3            -> origin/mlazos/hc-fixes3
2025-09-07T07:36:04.2640302Z  * [new branch]              mlazos/hc-fixes4            -> origin/mlazos/hc-fixes4
2025-09-07T07:36:04.2641567Z  * [new branch]              mlazos/hc-hf                -> origin/mlazos/hc-hf
2025-09-07T07:36:04.2642746Z  * [new branch]              mlazos/hc-mut               -> origin/mlazos/hc-mut
2025-09-07T07:36:04.2644069Z  * [new branch]              mlazos/hc10                 -> origin/mlazos/hc10
2025-09-07T07:36:04.2645284Z  * [new branch]              mlazos/hc11                 -> origin/mlazos/hc11
2025-09-07T07:36:04.2646500Z  * [new branch]              mlazos/hc12                 -> origin/mlazos/hc12
2025-09-07T07:36:04.2647679Z  * [new branch]              mlazos/hc13                 -> origin/mlazos/hc13
2025-09-07T07:36:04.2648963Z  * [new branch]              mlazos/hc14                 -> origin/mlazos/hc14
2025-09-07T07:36:04.2650178Z  * [new branch]              mlazos/hc15                 -> origin/mlazos/hc15
2025-09-07T07:36:04.2651440Z  * [new branch]              mlazos/hc2                  -> origin/mlazos/hc2
2025-09-07T07:36:04.2652631Z  * [new branch]              mlazos/hc4                  -> origin/mlazos/hc4
2025-09-07T07:36:04.2653866Z  * [new branch]              mlazos/hc5                  -> origin/mlazos/hc5
2025-09-07T07:36:04.2655078Z  * [new branch]              mlazos/hc6                  -> origin/mlazos/hc6
2025-09-07T07:36:04.2656300Z  * [new branch]              mlazos/hc7                  -> origin/mlazos/hc7
2025-09-07T07:36:04.2657484Z  * [new branch]              mlazos/hc8                  -> origin/mlazos/hc8
2025-09-07T07:36:04.2658889Z  * [new branch]              mlazos/hc9                  -> origin/mlazos/hc9
2025-09-07T07:36:04.2660134Z  * [new branch]              mlazos/hc_baseline2         -> origin/mlazos/hc_baseline2
2025-09-07T07:36:04.2661342Z  * [new branch]              mlazos/init-per-param       -> origin/mlazos/init-per-param
2025-09-07T07:36:04.2662561Z  * [new branch]              mlazos/init_per_param       -> origin/mlazos/init_per_param
2025-09-07T07:36:04.2663719Z  * [new branch]              mlazos/less-guards          -> origin/mlazos/less-guards
2025-09-07T07:36:04.2664956Z  * [new branch]              mlazos/lr-composibility     -> origin/mlazos/lr-composibility
2025-09-07T07:36:04.2666116Z  * [new branch]              mlazos/main                 -> origin/mlazos/main
2025-09-07T07:36:04.2667382Z  * [new branch]              mlazos/main-test-enablement -> origin/mlazos/main-test-enablement
2025-09-07T07:36:04.2668542Z  * [new branch]              mlazos/main2                -> origin/mlazos/main2
2025-09-07T07:36:04.2669799Z  * [new branch]              mlazos/mark-static-update   -> origin/mlazos/mark-static-update
2025-09-07T07:36:04.2671031Z  * [new branch]              mlazos/mcg                  -> origin/mlazos/mcg
2025-09-07T07:36:04.2672663Z  * [new branch]              mlazos/mcg2                 -> origin/mlazos/mcg2
2025-09-07T07:36:04.2673947Z  * [new branch]              mlazos/meta-guards          -> origin/mlazos/meta-guards
2025-09-07T07:36:04.2675542Z  * [new branch]              mlazos/mlazos/ck2           -> origin/mlazos/mlazos/ck2
2025-09-07T07:36:04.2676846Z  * [new branch]              mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam
2025-09-07T07:36:04.2678503Z  * [new branch]              mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup
2025-09-07T07:36:04.2679693Z  * [new branch]              mlazos/mod-fix              -> origin/mlazos/mod-fix
2025-09-07T07:36:04.2680938Z  * [new branch]              mlazos/mode-fix             -> origin/mlazos/mode-fix
2025-09-07T07:36:04.2682189Z  * [new branch]              mlazos/more-tests           -> origin/mlazos/more-tests
2025-09-07T07:36:04.2683404Z  * [new branch]              mlazos/no-cpp               -> origin/mlazos/no-cpp
2025-09-07T07:36:04.2684808Z  * [new branch]              mlazos/no-init-group-handling -> origin/mlazos/no-init-group-handling
2025-09-07T07:36:04.2685948Z  * [new branch]              mlazos/offsets              -> origin/mlazos/offsets
2025-09-07T07:36:04.2687123Z  * [new branch]              mlazos/opt-bench-exp2       -> origin/mlazos/opt-bench-exp2
2025-09-07T07:36:04.2688441Z  * [new branch]              mlazos/opt-incr             -> origin/mlazos/opt-incr
2025-09-07T07:36:04.2689653Z  * [new branch]              mlazos/proxy-ctors          -> origin/mlazos/proxy-ctors
2025-09-07T07:36:04.2691011Z  * [new branch]              mlazos/quant-fix            -> origin/mlazos/quant-fix
2025-09-07T07:36:04.2692265Z  * [new branch]              mlazos/resnet-fix           -> origin/mlazos/resnet-fix
2025-09-07T07:36:04.2693479Z  * [new branch]              mlazos/revert-inline        -> origin/mlazos/revert-inline
2025-09-07T07:36:04.2694705Z  * [new branch]              mlazos/rm-buf-names         -> origin/mlazos/rm-buf-names
2025-09-07T07:36:04.2695825Z  * [new branch]              mlazos/rm-code              -> origin/mlazos/rm-code
2025-09-07T07:36:04.2697055Z  * [new branch]              mlazos/rm-spam              -> origin/mlazos/rm-spam
2025-09-07T07:36:04.2698417Z  * [new branch]              mlazos/rtp                  -> origin/mlazos/rtp
2025-09-07T07:36:04.2699722Z  * [new branch]              mlazos/static-idx-dbg       -> origin/mlazos/static-idx-dbg
2025-09-07T07:36:04.2700941Z  * [new branch]              mlazos/static-inputs-log    -> origin/mlazos/static-inputs-log
2025-09-07T07:36:04.2702181Z  * [new branch]              mlazos/sub-param-fix        -> origin/mlazos/sub-param-fix
2025-09-07T07:36:04.2703561Z  * [new branch]              mlazos/td-fix2              -> origin/mlazos/td-fix2
2025-09-07T07:36:04.2704879Z  * [new branch]              mlazos/tensor-hasattr2      -> origin/mlazos/tensor-hasattr2
2025-09-07T07:36:04.2706070Z  * [new branch]              mlazos/test                 -> origin/mlazos/test
2025-09-07T07:36:04.2707256Z  * [new branch]              mlazos/tf-mode              -> origin/mlazos/tf-mode
2025-09-07T07:36:04.2708511Z  * [new branch]              mlazos/tf-mode-backup2      -> origin/mlazos/tf-mode-backup2
2025-09-07T07:36:04.2709728Z  * [new branch]              mlazos/tf-mode-reland       -> origin/mlazos/tf-mode-reland
2025-09-07T07:36:04.2711097Z  * [new branch]              mlazos/tf-mode-reland2      -> origin/mlazos/tf-mode-reland2
2025-09-07T07:36:04.2712394Z  * [new branch]              mlazos/tf-mode-reland3      -> origin/mlazos/tf-mode-reland3
2025-09-07T07:36:04.2713492Z  * [new branch]              mlazos/topo-fix             -> origin/mlazos/topo-fix
2025-09-07T07:36:04.2714706Z  * [new branch]              mlazos/triton-no-epi        -> origin/mlazos/triton-no-epi
2025-09-07T07:36:04.2715917Z  * [new branch]              mlazos/tune-proto           -> origin/mlazos/tune-proto
2025-09-07T07:36:04.2717109Z  * [new branch]              mlazos/tuple-fixes          -> origin/mlazos/tuple-fixes
2025-09-07T07:36:04.2718323Z  * [new branch]              mlazos/tuple-fixes2         -> origin/mlazos/tuple-fixes2
2025-09-07T07:36:04.2719609Z  * [new branch]              mlazos/tuple-handling       -> origin/mlazos/tuple-handling
2025-09-07T07:36:04.2720868Z  * [new branch]              mlazos/user-streams         -> origin/mlazos/user-streams
2025-09-07T07:36:04.2722615Z  * [new branch]              mlazos/vary-beta            -> origin/mlazos/vary-beta
2025-09-07T07:36:04.2723863Z  * [new branch]              mlazos/vary-beta2           -> origin/mlazos/vary-beta2
2025-09-07T07:36:04.2725157Z  * [new branch]              mlazos/weird-perf1          -> origin/mlazos/weird-perf1
2025-09-07T07:36:04.2726508Z  * [new branch]              mm_out_dtype_compile        -> origin/mm_out_dtype_compile
2025-09-07T07:36:04.2727763Z  * [new branch]              modify-setupvllm            -> origin/modify-setupvllm
2025-09-07T07:36:04.2729012Z  * [new branch]              module-shim                 -> origin/module-shim
2025-09-07T07:36:04.2730362Z  * [new branch]              move-theme-out-docker       -> origin/move-theme-out-docker
2025-09-07T07:36:04.2732093Z  * [new branch]              msaroufim/be1               -> origin/msaroufim/be1
2025-09-07T07:36:04.2733312Z  * [new branch]              msaroufim/cn_path           -> origin/msaroufim/cn_path
2025-09-07T07:36:04.2734556Z  * [new branch]              msaroufim/dtensorfusedadam  -> origin/msaroufim/dtensorfusedadam
2025-09-07T07:36:04.2735742Z  * [new branch]              msaroufim/reduce            -> origin/msaroufim/reduce
2025-09-07T07:36:04.2737522Z  * [new branch]              mtia/basic-cmake            -> origin/mtia/basic-cmake
2025-09-07T07:36:04.2738892Z  * [new branch]              muon_dev                    -> origin/muon_dev
2025-09-07T07:36:04.2740222Z  * [new branch]              muon_dev_1                  -> origin/muon_dev_1
2025-09-07T07:36:04.2741494Z  * [new branch]              nativert_num_outputs        -> origin/nativert_num_outputs
2025-09-07T07:36:04.2742849Z  * [new branch]              nativert_numoutputs         -> origin/nativert_numoutputs
2025-09-07T07:36:04.2744123Z  * [new branch]              new-modifiy-setupvllm       -> origin/new-modifiy-setupvllm
2025-09-07T07:36:04.2745357Z  * [new branch]              new-setupvllm               -> origin/new-setupvllm
2025-09-07T07:36:04.2746637Z  * [new branch]              new_zeros_dtype             -> origin/new_zeros_dtype
2025-09-07T07:36:04.2747982Z  * [new branch]              newtest-base                -> origin/newtest-base
2025-09-07T07:36:04.2749684Z  * [new branch]              ngimel/cat_perf1            -> origin/ngimel/cat_perf1
2025-09-07T07:36:04.2750819Z  * [new branch]              ngimel/einsum_fix           -> origin/ngimel/einsum_fix
2025-09-07T07:36:04.2752403Z  * [new branch]              ngimel/error_index_list     -> origin/ngimel/error_index_list
2025-09-07T07:36:04.2753547Z  * [new branch]              ngimel/fabric_check         -> origin/ngimel/fabric_check
2025-09-07T07:36:04.2754694Z  * [new branch]              ngimel/fabric_fix           -> origin/ngimel/fabric_fix
2025-09-07T07:36:04.2755989Z  * [new branch]              ngimel/fix_driver_init_error -> origin/ngimel/fix_driver_init_error
2025-09-07T07:36:04.2757483Z  * [new branch]              ngimel/fix_nccl_segment_seg -> origin/ngimel/fix_nccl_segment_seg
2025-09-07T07:36:04.2758932Z  * [new branch]              ngimel/gg_new               -> origin/ngimel/gg_new
2025-09-07T07:36:04.2760331Z  * [new branch]              ngimel/modeguard            -> origin/ngimel/modeguard
2025-09-07T07:36:04.2762287Z  * [new branch]              ngimel/multicast_fix        -> origin/ngimel/multicast_fix
2025-09-07T07:36:04.2763630Z  * [new branch]              ngimel/rocm_handle_type     -> origin/ngimel/rocm_handle_type
2025-09-07T07:36:04.2764921Z  * [new branch]              ngimel/symm_handle_fabric   -> origin/ngimel/symm_handle_fabric
2025-09-07T07:36:04.2766128Z  * [new branch]              ngimel/unbind_multimem      -> origin/ngimel/unbind_multimem
2025-09-07T07:36:04.2767398Z  * [new branch]              nightly                     -> origin/nightly
2025-09-07T07:36:04.2768767Z  * [new branch]              nmacchioni-patch-10         -> origin/nmacchioni-patch-10
2025-09-07T07:36:04.2770069Z  * [new branch]              nmacchioni-patch-7          -> origin/nmacchioni-patch-7
2025-09-07T07:36:04.2771452Z  * [new branch]              nmacchioni-patch-8          -> origin/nmacchioni-patch-8
2025-09-07T07:36:04.2772843Z  * [new branch]              nmacchioni-patch-9          -> origin/nmacchioni-patch-9
2025-09-07T07:36:04.2774634Z  * [new branch]              nullplay/fuse_matmul        -> origin/nullplay/fuse_matmul
2025-09-07T07:36:04.2775875Z  * [new branch]              nullplay_fuse_matmul        -> origin/nullplay_fuse_matmul
2025-09-07T07:36:04.2777134Z  * [new branch]              one-off                     -> origin/one-off
2025-09-07T07:36:04.2779532Z  * [new branch]              orig/release/1.10           -> origin/orig/release/1.10
2025-09-07T07:36:04.2780785Z  * [new branch]              orig/release/1.11           -> origin/orig/release/1.11
2025-09-07T07:36:04.2782034Z  * [new branch]              orig/release/1.12           -> origin/orig/release/1.12
2025-09-07T07:36:04.2783505Z  * [new branch]              orig/release/1.13           -> origin/orig/release/1.13
2025-09-07T07:36:04.2784781Z  * [new branch]              orig/release/1.6            -> origin/orig/release/1.6
2025-09-07T07:36:04.2786120Z  * [new branch]              orig/release/1.7            -> origin/orig/release/1.7
2025-09-07T07:36:04.2787370Z  * [new branch]              orig/release/1.8            -> origin/orig/release/1.8
2025-09-07T07:36:04.2788639Z  * [new branch]              orig/release/1.9            -> origin/orig/release/1.9
2025-09-07T07:36:04.2789901Z  * [new branch]              orig/release/2.0            -> origin/orig/release/2.0
2025-09-07T07:36:04.2791232Z  * [new branch]              orig/release/2.1            -> origin/orig/release/2.1
2025-09-07T07:36:04.2792503Z  * [new branch]              orig/release/2.2            -> origin/orig/release/2.2
2025-09-07T07:36:04.2793686Z  * [new branch]              orig/release/2.3            -> origin/orig/release/2.3
2025-09-07T07:36:04.2794878Z  * [new branch]              orig/release/2.4            -> origin/orig/release/2.4
2025-09-07T07:36:04.2796059Z  * [new branch]              orig/release/2.5            -> origin/orig/release/2.5
2025-09-07T07:36:04.2797279Z  * [new branch]              orig/release/2.6            -> origin/orig/release/2.6
2025-09-07T07:36:04.2798791Z  * [new branch]              orig/release/2.7            -> origin/orig/release/2.7
2025-09-07T07:36:04.2800431Z  * [new branch]              orig/release/2.8            -> origin/orig/release/2.8
2025-09-07T07:36:04.2802269Z  * [new branch]              oulgen/fx_graph             -> origin/oulgen/fx_graph
2025-09-07T07:36:04.2803772Z  * [new branch]              padded-tensor               -> origin/padded-tensor
2025-09-07T07:36:04.2805103Z  * [new branch]              pca2                        -> origin/pca2
2025-09-07T07:36:04.2806504Z  * [new branch]              pianpwk-patch-1             -> origin/pianpwk-patch-1
2025-09-07T07:36:04.2808297Z  * [new branch]              pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export
2025-09-07T07:36:04.2809560Z  * [new branch]              pianpwk/invalidate_fake_memo -> origin/pianpwk/invalidate_fake_memo
2025-09-07T07:36:04.2810584Z  * [new branch]              pianpwk/max_1_strides       -> origin/pianpwk/max_1_strides
2025-09-07T07:36:04.2811714Z  * [new branch]              pianpwk/maybe_guard_rel     -> origin/pianpwk/maybe_guard_rel
2025-09-07T07:36:04.2812803Z  * [new branch]              pianpwk/nonzero_memo        -> origin/pianpwk/nonzero_memo
2025-09-07T07:36:04.2814260Z  * [new branch]              pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better
2025-09-07T07:36:04.2815758Z  * [new branch]              pianpwk/oblivious_slice_forward -> origin/pianpwk/oblivious_slice_forward
2025-09-07T07:36:04.2816998Z  * [new branch]              pianpwk/oblivious_where     -> origin/pianpwk/oblivious_where
2025-09-07T07:36:04.2818281Z  * [new branch]              pianpwk/param_static_pgo    -> origin/pianpwk/param_static_pgo
2025-09-07T07:36:04.2819478Z  * [new branch]              pianpwk/pre_forward_hook    -> origin/pianpwk/pre_forward_hook
2025-09-07T07:36:04.2820840Z  * [new branch]              pianpwk/remove_guard_fail_break -> origin/pianpwk/remove_guard_fail_break
2025-09-07T07:36:04.2822014Z  * [new branch]              pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols
2025-09-07T07:36:04.2823178Z  * [new branch]              pianpwk/sym_tokens_draft    -> origin/pianpwk/sym_tokens_draft
2025-09-07T07:36:04.2824638Z  * [new branch]              pianpwk/test_pointwise_guard_or_false -> origin/pianpwk/test_pointwise_guard_or_false
2025-09-07T07:36:04.2825815Z  * [new branch]              pianpwk/test_slice_fake_impl -> origin/pianpwk/test_slice_fake_impl
2025-09-07T07:36:04.2827076Z  * [new branch]              pianpwk/totally_draft_sym_wrap -> origin/pianpwk/totally_draft_sym_wrap
2025-09-07T07:36:04.2828271Z  * [new branch]              pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last
2025-09-07T07:36:04.2829556Z  * [new branch]              pianpwk/unbacked_safe_conv1d -> origin/pianpwk/unbacked_safe_conv1d
2025-09-07T07:36:04.2830774Z  * [new branch]              pianpwk/unbacked_sdpa_flash -> origin/pianpwk/unbacked_sdpa_flash
2025-09-07T07:36:04.2832079Z  * [new branch]              pianpwk/unbacked_should_swap -> origin/pianpwk/unbacked_should_swap
2025-09-07T07:36:04.2833256Z  * [new branch]              pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2
2025-09-07T07:36:04.2834441Z  * [new branch]              pianpwk/unbacked_slice_binding -> origin/pianpwk/unbacked_slice_binding
2025-09-07T07:36:04.2835642Z  * [new branch]              pianpwk/unbacked_slice_forward -> origin/pianpwk/unbacked_slice_forward
2025-09-07T07:36:04.2836876Z  * [new branch]              pianpwk/user_symints        -> origin/pianpwk/user_symints
2025-09-07T07:36:04.2839040Z  * [new branch]              pianpwk/wan21_reshape       -> origin/pianpwk/wan21_reshape
2025-09-07T07:36:04.2840519Z  * [new branch]              pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer
2025-09-07T07:36:04.2841153Z  * [new branch]              pin-torchao                 -> origin/pin-torchao
2025-09-07T07:36:04.2842572Z  * [new branch]              piz/fall_back_missing_0716  -> origin/piz/fall_back_missing_0716
2025-09-07T07:36:04.2843680Z  * [new branch]              piz/improve_scatter_0808    -> origin/piz/improve_scatter_0808
2025-09-07T07:36:04.2844966Z  * [new branch]              pool-separate               -> origin/pool-separate
2025-09-07T07:36:04.2846226Z  * [new branch]              pr-156087                   -> origin/pr-156087
2025-09-07T07:36:04.2848061Z  * [new branch]              pr/131860                   -> origin/pr/131860
2025-09-07T07:36:04.2849351Z  * [new branch]              predispatch_to              -> origin/predispatch_to
2025-09-07T07:36:04.2850698Z  * [new branch]              pt-opt-cuda3                -> origin/pt-opt-cuda3
2025-09-07T07:36:04.2851974Z  * [new branch]              pyobjectslot                -> origin/pyobjectslot
2025-09-07T07:36:04.2854032Z  * [new branch]              python_compiled_autograd    -> origin/python_compiled_autograd
2025-09-07T07:36:04.2856226Z  * [new branch]              qchip/export-D54134695      -> origin/qchip/export-D54134695
2025-09-07T07:36:04.2857470Z  * [new branch]              quint-bits                  -> origin/quint-bits
2025-09-07T07:36:04.2859983Z  * [new branch]              release/1.10                -> origin/release/1.10
2025-09-07T07:36:04.2861252Z  * [new branch]              release/1.11                -> origin/release/1.11
2025-09-07T07:36:04.2862498Z  * [new branch]              release/1.12                -> origin/release/1.12
2025-09-07T07:36:04.2863719Z  * [new branch]              release/1.13                -> origin/release/1.13
2025-09-07T07:36:04.2864909Z  * [new branch]              release/1.4                 -> origin/release/1.4
2025-09-07T07:36:04.2865942Z  * [new branch]              release/1.4.1               -> origin/release/1.4.1
2025-09-07T07:36:04.2867160Z  * [new branch]              release/1.5                 -> origin/release/1.5
2025-09-07T07:36:04.2868434Z  * [new branch]              release/1.6                 -> origin/release/1.6
2025-09-07T07:36:04.2869678Z  * [new branch]              release/1.7                 -> origin/release/1.7
2025-09-07T07:36:04.2871057Z  * [new branch]              release/1.8                 -> origin/release/1.8
2025-09-07T07:36:04.2872183Z  * [new branch]              release/1.9                 -> origin/release/1.9
2025-09-07T07:36:04.2873420Z  * [new branch]              release/2.0                 -> origin/release/2.0
2025-09-07T07:36:04.2874714Z  * [new branch]              release/2.1                 -> origin/release/2.1
2025-09-07T07:36:04.2875975Z  * [new branch]              release/2.2                 -> origin/release/2.2
2025-09-07T07:36:04.2877526Z  * [new branch]              release/2.3                 -> origin/release/2.3
2025-09-07T07:36:04.2879209Z  * [new branch]              release/2.4                 -> origin/release/2.4
2025-09-07T07:36:04.2880848Z  * [new branch]              release/2.5                 -> origin/release/2.5
2025-09-07T07:36:04.2882238Z  * [new branch]              release/2.6                 -> origin/release/2.6
2025-09-07T07:36:04.2883528Z  * [new branch]              release/2.7                 -> origin/release/2.7
2025-09-07T07:36:04.2884796Z  * [new branch]              release/2.8                 -> origin/release/2.8
2025-09-07T07:36:04.2886106Z  * [new branch]              release_notes               -> origin/release_notes
2025-09-07T07:36:04.2887416Z  * [new branch]              remove-actionable-label     -> origin/remove-actionable-label
2025-09-07T07:36:04.2888649Z  * [new branch]              remove-ao                   -> origin/remove-ao
2025-09-07T07:36:04.2890160Z  * [new branch]              removedeprecatedvllmtest    -> origin/removedeprecatedvllmtest
2025-09-07T07:36:04.2891529Z  * [new branch]              replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836
2025-09-07T07:36:04.2892655Z  * [new branch]              replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248
2025-09-07T07:36:04.2893904Z  * [new branch]              replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324
2025-09-07T07:36:04.2895214Z  * [new branch]              replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020
2025-09-07T07:36:04.2896438Z  * [new branch]              replace-pytorch-labs-20250812-204125 -> origin/replace-pytorch-labs-20250812-204125
2025-09-07T07:36:04.2897853Z  * [new branch]              replace-pytorch-labs-20250812-205624 -> origin/replace-pytorch-labs-20250812-205624
2025-09-07T07:36:04.2900593Z  * [new branch]              revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head
2025-09-07T07:36:04.2903331Z  * [new branch]              revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head
2025-09-07T07:36:04.2906190Z  * [new branch]              revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head
2025-09-07T07:36:04.2907562Z  * [new branch]              revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_
2025-09-07T07:36:04.2908655Z  * [new branch]              rocm-monitoring             -> origin/rocm-monitoring
2025-09-07T07:36:04.2910458Z  * [new branch]              ruisi/relax_memory          -> origin/ruisi/relax_memory
2025-09-07T07:36:04.2911776Z  * [new branch]              run-torchbench-smoke-test-h100 -> origin/run-torchbench-smoke-test-h100
2025-09-07T07:36:04.2913677Z  * [new branch]              ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures
2025-09-07T07:36:04.2914741Z  * [new branch]              ryanguo99/fix-closure-var   -> origin/ryanguo99/fix-closure-var
2025-09-07T07:36:04.2916474Z  * [new branch]              rzou/faketensor_bench       -> origin/rzou/faketensor_bench
2025-09-07T07:36:04.2917597Z  * [new branch]              rzou/njt                    -> origin/rzou/njt
2025-09-07T07:36:04.2918783Z  * [new branch]              rzou/pca                    -> origin/rzou/pca
2025-09-07T07:36:04.2919928Z  * [new branch]              rzou/realprop               -> origin/rzou/realprop
2025-09-07T07:36:04.2921121Z  * [new branch]              rzou/setup_context          -> origin/rzou/setup_context
2025-09-07T07:36:04.2923069Z  * [new branch]              sanchitintel/refactor_aten_int8_woq_gemm -> origin/sanchitintel/refactor_aten_int8_woq_gemm
2025-09-07T07:36:04.2924329Z  * [new branch]              sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm
2025-09-07T07:36:04.2925558Z  * [new branch]              sapling-pr-archive-SS-JIA   -> origin/sapling-pr-archive-SS-JIA
2025-09-07T07:36:04.2926775Z  * [new branch]              save                        -> origin/save
2025-09-07T07:36:04.2928600Z  * [new branch]              sdym/2.5.1                  -> origin/sdym/2.5.1
2025-09-07T07:36:04.2929926Z  * [new branch]              seemethere-patch-1          -> origin/seemethere-patch-1
2025-09-07T07:36:04.2931251Z  * [new branch]              setupvllm                   -> origin/setupvllm
2025-09-07T07:36:04.2932656Z  * [new branch]              share_and_pin_fork          -> origin/share_and_pin_fork
2025-09-07T07:36:04.2934489Z  * [new branch]              shengf/fx-xform-perf        -> origin/shengf/fx-xform-perf
2025-09-07T07:36:04.2935716Z  * [new branch]              shikaili_fp8_allgather      -> origin/shikaili_fp8_allgather
2025-09-07T07:36:04.2937065Z  * [new branch]              shoumikhin-patch-1          -> origin/shoumikhin-patch-1
2025-09-07T07:36:04.2938588Z  * [new branch]              shoumikhin-patch-12         -> origin/shoumikhin-patch-12
2025-09-07T07:36:04.2940131Z  * [new branch]              simplify-fq-per-channel     -> origin/simplify-fq-per-channel
2025-09-07T07:36:04.2941470Z  * [new branch]              solve-accuracy-fix          -> origin/solve-accuracy-fix
2025-09-07T07:36:04.2943130Z  * [new branch]              soulitzer/stash-tls-ac      -> origin/soulitzer/stash-tls-ac
2025-09-07T07:36:04.2944895Z  * [new branch]              sqzhang/flight4             -> origin/sqzhang/flight4
2025-09-07T07:36:04.2946128Z  * [new branch]              sqzhang/flight4plus         -> origin/sqzhang/flight4plus
2025-09-07T07:36:04.2947866Z  * [new branch]              sraikund/record_funct_test  -> origin/sraikund/record_funct_test
2025-09-07T07:36:04.2949575Z  * [new branch]              sraikund16/test             -> origin/sraikund16/test
2025-09-07T07:36:04.2951431Z  * [new branch]              stablize-compilation-time   -> origin/stablize-compilation-time
2025-09-07T07:36:04.2952693Z  * [new branch]              standalone-templates        -> origin/standalone-templates
2025-09-07T07:36:04.2954104Z  * [new branch]              standalone_package_weights  -> origin/standalone_package_weights
2025-09-07T07:36:04.2955240Z  * [new branch]              starterTaskUpdate           -> origin/starterTaskUpdate
2025-09-07T07:36:04.2956475Z  * [new branch]              subgraph_fuse               -> origin/subgraph_fuse
2025-09-07T07:36:04.2957845Z  * [new branch]              support-uv-in-collect_env   -> origin/support-uv-in-collect_env
2025-09-07T07:36:04.2959070Z  * [new branch]              sve-poc                     -> origin/sve-poc
2025-09-07T07:36:04.2960332Z  * [new branch]              svekars-patch-1             -> origin/svekars-patch-1
2025-09-07T07:36:04.2961773Z  * [new branch]              switch-bn                   -> origin/switch-bn
2025-09-07T07:36:04.2963065Z  * [new branch]              sympy-bottleneck-repro      -> origin/sympy-bottleneck-repro
2025-09-07T07:36:04.2964870Z  * [new branch]              tenpercent/ck_rocm_ci_v3    -> origin/tenpercent/ck_rocm_ci_v3
2025-09-07T07:36:04.2966233Z  * [new branch]              tensordict_integration      -> origin/tensordict_integration
2025-09-07T07:36:04.2967456Z  * [new branch]              test-7054                   -> origin/test-7054
2025-09-07T07:36:04.2968935Z  * [new branch]              test-move-conda-builds      -> origin/test-move-conda-builds
2025-09-07T07:36:04.2970348Z  * [new branch]              test-myst-markdown-docstring -> origin/test-myst-markdown-docstring
2025-09-07T07:36:04.2971551Z  * [new branch]              test-old                    -> origin/test-old
2025-09-07T07:36:04.2972866Z  * [new branch]              test-vec-migration-internally -> origin/test-vec-migration-internally
2025-09-07T07:36:04.2974513Z  * [new branch]              test/bmm_heur               -> origin/test/bmm_heur
2025-09-07T07:36:04.2975685Z  * [new branch]              test/inductor               -> origin/test/inductor
2025-09-07T07:36:04.2977452Z  * [new branch]              tianren/flex_paged_attn_fix -> origin/tianren/flex_paged_attn_fix
2025-09-07T07:36:04.2978712Z  * [new branch]              tianren/flex_paged_attn_fix_temp -> origin/tianren/flex_paged_attn_fix_temp
2025-09-07T07:36:04.2979783Z  * [new branch]              tianren/test                -> origin/tianren/test
2025-09-07T07:36:04.2981080Z  * [new branch]              tidy_performance_cyy        -> origin/tidy_performance_cyy
2025-09-07T07:36:04.2982371Z  * [new branch]              torchtitan_ep               -> origin/torchtitan_ep
2025-09-07T07:36:04.2983844Z  * [new branch]              trace_fsdp_torchtune_lora   -> origin/trace_fsdp_torchtune_lora
2025-09-07T07:36:04.2985044Z  * [new branch]              traceable_fsdp_unit_tests   -> origin/traceable_fsdp_unit_tests
2025-09-07T07:36:04.2986362Z  * [new branch]              tree_loop_vec_base          -> origin/tree_loop_vec_base
2025-09-07T07:36:04.2987667Z  * [new branch]              tree_vec_base               -> origin/tree_vec_base
2025-09-07T07:36:04.2989002Z  * [new branch]              triton-update               -> origin/triton-update
2025-09-07T07:36:04.2990229Z  * [new branch]              triton_kernel               -> origin/triton_kernel
2025-09-07T07:36:04.2991406Z  * [new branch]              triton_kernel_perf          -> origin/triton_kernel_perf
2025-09-07T07:36:04.2992662Z  * [new branch]              tt_pkg_1908                 -> origin/tt_pkg_1908
2025-09-07T07:36:04.2994020Z  * [new branch]              tweak-transformer-dependabot -> origin/tweak-transformer-dependabot
2025-09-07T07:36:04.2995189Z  * [new branch]              type_dec                    -> origin/type_dec
2025-09-07T07:36:04.2996598Z  * [new branch]              udate-sphinx-dependancies   -> origin/udate-sphinx-dependancies
2025-09-07T07:36:04.2998837Z  * [new branch]              update-audio-commit-hash/16818882925-1712-1 -> origin/update-audio-commit-hash/16818882925-1712-1
2025-09-07T07:36:04.3000070Z  * [new branch]              update-audio-commit-hash/16895560422-1720-1 -> origin/update-audio-commit-hash/16895560422-1720-1
2025-09-07T07:36:04.3001333Z  * [new branch]              update-audio-commit-hash/16924174496-1738-1 -> origin/update-audio-commit-hash/16924174496-1738-1
2025-09-07T07:36:04.3002576Z  * [new branch]              update-audio-commit-hash/17002010821-1749-1 -> origin/update-audio-commit-hash/17002010821-1749-1
2025-09-07T07:36:04.3007481Z  * [new branch]              update-audio-commit-hash/17056004427-1766-1 -> origin/update-audio-commit-hash/17056004427-1766-1
2025-09-07T07:36:04.3008949Z  * [new branch]              update-audio-commit-hash/17085054029-1767-1 -> origin/update-audio-commit-hash/17085054029-1767-1
2025-09-07T07:36:04.3010430Z  * [new branch]              update-audio-commit-hash/17142507405-1771-1 -> origin/update-audio-commit-hash/17142507405-1771-1
2025-09-07T07:36:04.3012008Z  * [new branch]              update-audio-commit-hash/17168762740-1773-1 -> origin/update-audio-commit-hash/17168762740-1773-1
2025-09-07T07:36:04.3013240Z  * [new branch]              update-audio-commit-hash/17311174639-1780-1 -> origin/update-audio-commit-hash/17311174639-1780-1
2025-09-07T07:36:04.3014426Z  * [new branch]              update-audio-commit-hash/17336898740-1781-1 -> origin/update-audio-commit-hash/17336898740-1781-1
2025-09-07T07:36:04.3015622Z  * [new branch]              update-audio-commit-hash/17389727684-1786-1 -> origin/update-audio-commit-hash/17389727684-1786-1
2025-09-07T07:36:04.3016812Z  * [new branch]              update-audio-commit-hash/17449538142-1790-1 -> origin/update-audio-commit-hash/17449538142-1790-1
2025-09-07T07:36:04.3018107Z  * [new branch]              update-audio-commit-hash/17507351808-1794-1 -> origin/update-audio-commit-hash/17507351808-1794-1
2025-09-07T07:36:04.3019344Z  * [new branch]              update-dynamic-shapes-doc   -> origin/update-dynamic-shapes-doc
2025-09-07T07:36:04.3021180Z  * [new branch]              update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1
2025-09-07T07:36:04.3022848Z  * [new branch]              update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2
2025-09-07T07:36:04.3024541Z  * [new branch]              update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1
2025-09-07T07:36:04.3026324Z  * [new branch]              update-vllm-commit-hash/16737365217-1704-1 -> origin/update-vllm-commit-hash/16737365217-1704-1
2025-09-07T07:36:04.3027469Z  * [new branch]              update-vllm-commit-hash/16843157111-1713-1 -> origin/update-vllm-commit-hash/16843157111-1713-1
2025-09-07T07:36:04.3028534Z  * [new branch]              update-vllm-commit-hash/16855312394-1714-1 -> origin/update-vllm-commit-hash/16855312394-1714-1
2025-09-07T07:36:04.3029660Z  * [new branch]              update-vllm-commit-hash/16924174496-1738-1 -> origin/update-vllm-commit-hash/16924174496-1738-1
2025-09-07T07:36:04.3030766Z  * [new branch]              update-vllm-commit-hash/16952608705-1745-1 -> origin/update-vllm-commit-hash/16952608705-1745-1
2025-09-07T07:36:04.3032163Z  * [new branch]              update-vllm-commit-hash/16979836546-1748-1 -> origin/update-vllm-commit-hash/16979836546-1748-1
2025-09-07T07:36:04.3033714Z  * [new branch]              update-vllm-commit-hash/17014576881-1756-1 -> origin/update-vllm-commit-hash/17014576881-1756-1
2025-09-07T07:36:04.3035207Z  * [new branch]              update-vllm-commit-hash/17027830869-1761-1 -> origin/update-vllm-commit-hash/17027830869-1761-1
2025-09-07T07:36:04.3036468Z  * [new branch]              update-vllm-commit-hash/17056004427-1766-1 -> origin/update-vllm-commit-hash/17056004427-1766-1
2025-09-07T07:36:04.3037676Z  * [new branch]              update-vllm-commit-hash/17085054029-1767-1 -> origin/update-vllm-commit-hash/17085054029-1767-1
2025-09-07T07:36:04.3038978Z  * [new branch]              update-vllm-commit-hash/17113610216-1768-1 -> origin/update-vllm-commit-hash/17113610216-1768-1
2025-09-07T07:36:04.3040680Z  * [new branch]              update-vllm-commit-hash/17142507405-1771-1 -> origin/update-vllm-commit-hash/17142507405-1771-1
2025-09-07T07:36:04.3041663Z  * [new branch]              update-vllm-commit-hash/17181878974-1774-1 -> origin/update-vllm-commit-hash/17181878974-1774-1
2025-09-07T07:36:04.3043133Z  * [new branch]              update-vllm-commit-hash/17311174639-1780-1 -> origin/update-vllm-commit-hash/17311174639-1780-1
2025-09-07T07:36:04.3044534Z  * [new branch]              update-vllm-commit-hash/17336898740-1781-1 -> origin/update-vllm-commit-hash/17336898740-1781-1
2025-09-07T07:36:04.3045833Z  * [new branch]              update-vllm-commit-hash/17364352302-1785-1 -> origin/update-vllm-commit-hash/17364352302-1785-1
2025-09-07T07:36:04.3047030Z  * [new branch]              update-vllm-commit-hash/17389727684-1786-1 -> origin/update-vllm-commit-hash/17389727684-1786-1
2025-09-07T07:36:04.3048291Z  * [new branch]              update-vllm-commit-hash/17449538142-1790-1 -> origin/update-vllm-commit-hash/17449538142-1790-1
2025-09-07T07:36:04.3049521Z  * [new branch]              update-vllm-commit-hash/17480069797-1791-1 -> origin/update-vllm-commit-hash/17480069797-1791-1
2025-09-07T07:36:04.3050638Z  * [new branch]              update-vllm-commit-hash/17507351808-1794-1 -> origin/update-vllm-commit-hash/17507351808-1794-1
2025-09-07T07:36:04.3052390Z  * [new branch]              update-xla-commit-hash/16873912760-198-1 -> origin/update-xla-commit-hash/16873912760-198-1
2025-09-07T07:36:04.3053562Z  * [new branch]              update-xla-commit-hash/17034266655-199-1 -> origin/update-xla-commit-hash/17034266655-199-1
2025-09-07T07:36:04.3054716Z  * [new branch]              update-xla-commit-hash/17202464405-200-1 -> origin/update-xla-commit-hash/17202464405-200-1
2025-09-07T07:36:04.3055973Z  * [new branch]              update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388
2025-09-07T07:36:04.3057193Z  * [new branch]              update_executorch_pin       -> origin/update_executorch_pin
2025-09-07T07:36:04.3058654Z  * [new branch]              update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736
2025-09-07T07:36:04.3059989Z  * [new branch]              update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173
2025-09-07T07:36:04.3061278Z  * [new branch]              update_slow_tests_1752478971 -> origin/update_slow_tests_1752478971
2025-09-07T07:36:04.3062522Z  * [new branch]              update_slow_tests_1755502951 -> origin/update_slow_tests_1755502951
2025-09-07T07:36:04.3063786Z  * [new branch]              update_slow_tests_1756107664 -> origin/update_slow_tests_1756107664
2025-09-07T07:36:04.3065573Z  * [new branch]              update_submodule_FBGEMM     -> origin/update_submodule_FBGEMM
2025-09-07T07:36:04.3066889Z  * [new branch]              update_submodule_kineto     -> origin/update_submodule_kineto
2025-09-07T07:36:04.3068178Z  * [new branch]              update_submodule_tensorpipe -> origin/update_submodule_tensorpipe
2025-09-07T07:36:04.3069584Z  * [new branch]              v0.1.2                      -> origin/v0.1.2
2025-09-07T07:36:04.3070976Z  * [new branch]              v1.0.1                      -> origin/v1.0.1
2025-09-07T07:36:04.3072341Z  * [new branch]              v1.0.3                      -> origin/v1.0.3
2025-09-07T07:36:04.3073833Z  * [new branch]              v1.1.0                      -> origin/v1.1.0
2025-09-07T07:36:04.3075203Z  * [new branch]              v1.2.0                      -> origin/v1.2.0
2025-09-07T07:36:04.3076555Z  * [new branch]              v1.3.0                      -> origin/v1.3.0
2025-09-07T07:36:04.3077907Z  * [new branch]              v1.3.1                      -> origin/v1.3.1
2025-09-07T07:36:04.3079238Z  * [new branch]              validate_fn                 -> origin/validate_fn
2025-09-07T07:36:04.3080663Z  * [new branch]              validations_2.6             -> origin/validations_2.6
2025-09-07T07:36:04.3082118Z  * [new branch]              validations_2.8             -> origin/validations_2.8
2025-09-07T07:36:04.3084021Z  * [new branch]              viable/strict               -> origin/viable/strict
2025-09-07T07:36:04.3085221Z  * [new branch]              vllmbuildci                 -> origin/vllmbuildci
2025-09-07T07:36:04.3086629Z  * [new branch]              vllmpin                     -> origin/vllmpin
2025-09-07T07:36:04.3088819Z  * [new branch]              wdvr/conda_devcontainer     -> origin/wdvr/conda_devcontainer
2025-09-07T07:36:04.3089952Z  * [new branch]              wdvr/iss_145259             -> origin/wdvr/iss_145259
2025-09-07T07:36:04.3091394Z  * [new branch]              weight_sharing_cpp          -> origin/weight_sharing_cpp
2025-09-07T07:36:04.3093461Z  * [new branch]              whc/flight4                 -> origin/whc/flight4
2025-09-07T07:36:04.3094613Z  * [new branch]              whc/flight51                -> origin/whc/flight51
2025-09-07T07:36:04.3095777Z  * [new branch]              whc/flight53                -> origin/whc/flight53
2025-09-07T07:36:04.3097060Z  * [new branch]              whc/stage2                  -> origin/whc/stage2
2025-09-07T07:36:04.3098220Z  * [new branch]              whc/uneven                  -> origin/whc/uneven
2025-09-07T07:36:04.3099667Z  * [new branch]              whc/uneven-merge            -> origin/whc/uneven-merge
2025-09-07T07:36:04.3100970Z  * [new branch]              win_warnings                -> origin/win_warnings
2025-09-07T07:36:04.3102225Z  * [new branch]              windows_libtorch_free       -> origin/windows_libtorch_free
2025-09-07T07:36:04.3103587Z  * [new branch]              workonoldcommit             -> origin/workonoldcommit
2025-09-07T07:36:04.3105135Z  * [new branch]              wychi-autotune-prune-configs-by-shared-mem -> origin/wychi-autotune-prune-configs-by-shared-mem
2025-09-07T07:36:04.3106804Z  * [new branch]              xmfan/ca_0516               -> origin/xmfan/ca_0516
2025-09-07T07:36:04.3107943Z  * [new branch]              xmfan/ca_1051b93192         -> origin/xmfan/ca_1051b93192
2025-09-07T07:36:04.3109314Z  * [new branch]              xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8
2025-09-07T07:36:04.3110275Z  * [new branch]              xmfan/ca_5a2be192d1         -> origin/xmfan/ca_5a2be192d1
2025-09-07T07:36:04.3111360Z  * [new branch]              xmfan/ca_9d59b516e9         -> origin/xmfan/ca_9d59b516e9
2025-09-07T07:36:04.3112535Z  * [new branch]              xmfan/ca_api                -> origin/xmfan/ca_api
2025-09-07T07:36:04.3113655Z  * [new branch]              xmfan/ca_apr8               -> origin/xmfan/ca_apr8
2025-09-07T07:36:04.3115051Z  * [new branch]              xmfan/ca_base               -> origin/xmfan/ca_base
2025-09-07T07:36:04.3116637Z  * [new branch]              xmfan/ca_cudagraphs         -> origin/xmfan/ca_cudagraphs
2025-09-07T07:36:04.3117834Z  * [new branch]              xmfan/ca_dynamic            -> origin/xmfan/ca_dynamic
2025-09-07T07:36:04.3119082Z  * [new branch]              xmfan/ca_fix_dyn            -> origin/xmfan/ca_fix_dyn
2025-09-07T07:36:04.3120275Z  * [new branch]              xmfan/ca_fix_lowering       -> origin/xmfan/ca_fix_lowering
2025-09-07T07:36:04.3121559Z  * [new branch]              xmfan/ca_fix_polyfills      -> origin/xmfan/ca_fix_polyfills
2025-09-07T07:36:04.3122646Z  * [new branch]              xmfan/ca_jan3               -> origin/xmfan/ca_jan3
2025-09-07T07:36:04.3124274Z  * [new branch]              xmfan/ca_jun18              -> origin/xmfan/ca_jun18
2025-09-07T07:36:04.3125531Z  * [new branch]              xmfan/ca_jun24              -> origin/xmfan/ca_jun24
2025-09-07T07:36:04.3126721Z  * [new branch]              xmfan/ca_mem_base           -> origin/xmfan/ca_mem_base
2025-09-07T07:36:04.3127889Z  * [new branch]              xmfan/ca_mem_fix            -> origin/xmfan/ca_mem_fix
2025-09-07T07:36:04.3129118Z  * [new branch]              xmfan/ca_memory_fix         -> origin/xmfan/ca_memory_fix
2025-09-07T07:36:04.3130299Z  * [new branch]              xmfan/ca_memory_fix_rebased -> origin/xmfan/ca_memory_fix_rebased
2025-09-07T07:36:04.3131614Z  * [new branch]              xmfan/ca_memory_fix_rebased2 -> origin/xmfan/ca_memory_fix_rebased2
2025-09-07T07:36:04.3132751Z  * [new branch]              xmfan/ca_move_to_cuda       -> origin/xmfan/ca_move_to_cuda
2025-09-07T07:36:04.3133941Z  * [new branch]              xmfan/ca_nested             -> origin/xmfan/ca_nested
2025-09-07T07:36:04.3135161Z  * [new branch]              xmfan/ca_overhead           -> origin/xmfan/ca_overhead
2025-09-07T07:36:04.3136378Z  * [new branch]              xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451
2025-09-07T07:36:04.3137524Z  * [new branch]              xmfan/ca_scalar             -> origin/xmfan/ca_scalar
2025-09-07T07:36:04.3138839Z  * [new branch]              xmfan/ca_subclass_mem_fix   -> origin/xmfan/ca_subclass_mem_fix
2025-09-07T07:36:04.3140088Z  * [new branch]              xmfan/ca_warm_mem           -> origin/xmfan/ca_warm_mem
2025-09-07T07:36:04.3141496Z  * [new branch]              xmfan/ca_warm_mem_base      -> origin/xmfan/ca_warm_mem_base
2025-09-07T07:36:04.3142738Z  * [new branch]              xmfan/cacu_jun18            -> origin/xmfan/cacu_jun18
2025-09-07T07:36:04.3143879Z  * [new branch]              xmfan/cacu_jun19            -> origin/xmfan/cacu_jun19
2025-09-07T07:36:04.3145154Z  * [new branch]              xmfan/cacu_jun4             -> origin/xmfan/cacu_jun4
2025-09-07T07:36:04.3146536Z  * [new branch]              xmfan/cacu_may27            -> origin/xmfan/cacu_may27
2025-09-07T07:36:04.3147902Z  * [new branch]              xmfan/disable_duck_shape    -> origin/xmfan/disable_duck_shape
2025-09-07T07:36:04.3149312Z  * [new branch]              xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough
2025-09-07T07:36:04.3150590Z  * [new branch]              xmfan/issue_123374          -> origin/xmfan/issue_123374
2025-09-07T07:36:04.3152001Z  * [new branch]              xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9
2025-09-07T07:36:04.3153698Z  * [new branch]              xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9
2025-09-07T07:36:04.3154812Z  * [new branch]              xmfan/segfault_test         -> origin/xmfan/segfault_test
2025-09-07T07:36:04.3156022Z  * [new branch]              xmfan/single_step           -> origin/xmfan/single_step
2025-09-07T07:36:04.3157232Z  * [new branch]              xmfan/sth_0829              -> origin/xmfan/sth_0829
2025-09-07T07:36:04.3158649Z  * [new branch]              xmfan/test                  -> origin/xmfan/test
2025-09-07T07:36:04.3160501Z  * [new branch]              yguo/debug-0226-constexpr   -> origin/yguo/debug-0226-constexpr
2025-09-07T07:36:04.3161622Z  * [new branch]              yguo/new_latest_changes     -> origin/yguo/new_latest_changes
2025-09-07T07:36:04.3162753Z  * [new branch]              yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes
2025-09-07T07:36:04.3164010Z  * [new branch]              yihan_quantization          -> origin/yihan_quantization
2025-09-07T07:36:04.3166239Z  * [new branch]              yiming/add_jit_trace_benchmark -> origin/yiming/add_jit_trace_benchmark
2025-09-07T07:36:04.3167406Z  * [new branch]              yiming/add_nativert_benchmark -> origin/yiming/add_nativert_benchmark
2025-09-07T07:36:04.3168506Z  * [new branch]              yiming/bootcamp             -> origin/yiming/bootcamp
2025-09-07T07:36:04.3170276Z  * [new branch]              zainr/canary-test           -> origin/zainr/canary-test
2025-09-07T07:36:04.3171931Z  * [new branch]              zainr/cleanup-gh-runners    -> origin/zainr/cleanup-gh-runners
2025-09-07T07:36:04.3173049Z  * [new branch]              zainr/git-push-v2           -> origin/zainr/git-push-v2
2025-09-07T07:36:04.3174236Z  * [new branch]              zainr/pull-migration-c      -> origin/zainr/pull-migration-c
2025-09-07T07:36:04.3175523Z  * [new branch]              zainr/test                  -> origin/zainr/test
2025-09-07T07:36:04.3176705Z  * [new branch]              zainr/test2                 -> origin/zainr/test2
2025-09-07T07:36:04.3177789Z  * [new branch]              zainr/unstable              -> origin/zainr/unstable
2025-09-07T07:36:04.3178941Z  * [new branch]              zainr/unstable-xla          -> origin/zainr/unstable-xla
2025-09-07T07:36:04.3180322Z  * [new branch]              zasdfgbnm-patch-3           -> origin/zasdfgbnm-patch-3
2025-09-07T07:36:04.3181608Z  * [new branch]              zb2p                        -> origin/zb2p
2025-09-07T07:36:04.3183315Z  * [new branch]              zero_grad_optimization      -> origin/zero_grad_optimization
2025-09-07T07:36:04.3184699Z  * [new branch]              zeros-and-scatter-part2     -> origin/zeros-and-scatter-part2
2025-09-07T07:36:04.3186822Z  * [new branch]              zhxchen17/scratch/0         -> origin/zhxchen17/scratch/0
2025-09-07T07:36:04.3188671Z  * [new branch]              zhxhcen17/moodycamel        -> origin/zhxhcen17/moodycamel
2025-09-07T07:36:04.3190381Z  * [new branch]              zxiiro/main                 -> origin/zxiiro/main
2025-09-07T07:36:04.3191642Z  * [new tag]                 bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug
2025-09-07T07:36:04.3192698Z  * [new tag]                 ci/binaries/77164           -> ci/binaries/77164
2025-09-07T07:36:04.3193841Z  * [new tag]                 ciflow/binaries/156049      -> ciflow/binaries/156049
2025-09-07T07:36:04.3194553Z  * [new tag]                 ciflow/binaries/156712      -> ciflow/binaries/156712
2025-09-07T07:36:04.3195382Z  * [new tag]                 ciflow/binaries/157432      -> ciflow/binaries/157432
2025-09-07T07:36:04.3196206Z  * [new tag]                 ciflow/binaries/157685      -> ciflow/binaries/157685
2025-09-07T07:36:04.3196926Z  * [new tag]                 ciflow/binaries/157689      -> ciflow/binaries/157689
2025-09-07T07:36:04.3197751Z  * [new tag]                 ciflow/binaries/158104      -> ciflow/binaries/158104
2025-09-07T07:36:04.3198619Z  * [new tag]                 ciflow/binaries/160229      -> ciflow/binaries/160229
2025-09-07T07:36:04.3199502Z  * [new tag]                 ciflow/binaries/160720      -> ciflow/binaries/160720
2025-09-07T07:36:04.3200372Z  * [new tag]                 ciflow/binaries/162080      -> ciflow/binaries/162080
2025-09-07T07:36:04.3201022Z  * [new tag]                 ciflow/binaries/162329      -> ciflow/binaries/162329
2025-09-07T07:36:04.3202152Z  * [new tag]                 ciflow/binaries_libtorch/156049 -> ciflow/binaries_libtorch/156049
2025-09-07T07:36:04.3203021Z  * [new tag]                 ciflow/binaries_libtorch/156711 -> ciflow/binaries_libtorch/156711
2025-09-07T07:36:04.3204018Z  * [new tag]                 ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432
2025-09-07T07:36:04.3204972Z  * [new tag]                 ciflow/binaries_wheel/156049 -> ciflow/binaries_wheel/156049
2025-09-07T07:36:04.3205721Z  * [new tag]                 ciflow/binaries_wheel/156711 -> ciflow/binaries_wheel/156711
2025-09-07T07:36:04.3207115Z  * [new tag]                 ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432
2025-09-07T07:36:04.3207400Z  * [new tag]                 ciflow/binaries_wheel/162136 -> ciflow/binaries_wheel/162136
2025-09-07T07:36:04.3208123Z  * [new tag]                 ciflow/binaries_wheel/162252 -> ciflow/binaries_wheel/162252
2025-09-07T07:36:04.3209022Z  * [new tag]                 ciflow/binaries_wheel/162325 -> ciflow/binaries_wheel/162325
2025-09-07T07:36:04.3210026Z  * [new tag]                 ciflow/h100-distributed/156703 -> ciflow/h100-distributed/156703
2025-09-07T07:36:04.3210969Z  * [new tag]                 ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635
2025-09-07T07:36:04.3211684Z  * [new tag]                 ciflow/h100-symm-mem/161984 -> ciflow/h100-symm-mem/161984
2025-09-07T07:36:04.3212550Z  * [new tag]                 ciflow/h100-symm-mem/162003 -> ciflow/h100-symm-mem/162003
2025-09-07T07:36:04.3213662Z  * [new tag]                 ciflow/h100-symm-mem/162011 -> ciflow/h100-symm-mem/162011
2025-09-07T07:36:04.3214034Z  * [new tag]                 ciflow/h100-symm-mem/162026 -> ciflow/h100-symm-mem/162026
2025-09-07T07:36:04.3215543Z  * [new tag]                 ciflow/h100-symm-mem/162033 -> ciflow/h100-symm-mem/162033
2025-09-07T07:36:04.3216365Z  * [new tag]                 ciflow/h100-symm-mem/162040 -> ciflow/h100-symm-mem/162040
2025-09-07T07:36:04.3217083Z  * [new tag]                 ciflow/h100-symm-mem/162041 -> ciflow/h100-symm-mem/162041
2025-09-07T07:36:04.3217992Z  * [new tag]                 ciflow/h100-symm-mem/162142 -> ciflow/h100-symm-mem/162142
2025-09-07T07:36:04.3218776Z  * [new tag]                 ciflow/h100-symm-mem/162150 -> ciflow/h100-symm-mem/162150
2025-09-07T07:36:04.3219595Z  * [new tag]                 ciflow/h100-symm-mem/162243 -> ciflow/h100-symm-mem/162243
2025-09-07T07:36:04.3220637Z  * [new tag]                 ciflow/h100-symm-mem/162320 -> ciflow/h100-symm-mem/162320
2025-09-07T07:36:04.3221638Z  * [new tag]                 ciflow/h100/159158          -> ciflow/h100/159158
2025-09-07T07:36:04.3222831Z  * [new tag]                 ciflow/h100/160480          -> ciflow/h100/160480
2025-09-07T07:36:04.3223695Z  * [new tag]                 ciflow/h100/161749          -> ciflow/h100/161749
2025-09-07T07:36:04.3224606Z  * [new tag]                 ciflow/h100/162022          -> ciflow/h100/162022
2025-09-07T07:36:04.3225449Z  * [new tag]                 ciflow/h100/162278          -> ciflow/h100/162278
2025-09-07T07:36:04.3226671Z  * [new tag]                 ciflow/inductor-perf-test-nightly-rocm/156592 -> ciflow/inductor-perf-test-nightly-rocm/156592
2025-09-07T07:36:04.3227764Z  * [new tag]                 ciflow/inductor-perf-test-nightly/156592 -> ciflow/inductor-perf-test-nightly/156592
2025-09-07T07:36:04.3228814Z  * [new tag]                 ciflow/inductor-periodic/162063 -> ciflow/inductor-periodic/162063
2025-09-07T07:36:04.3229516Z  * [new tag]                 ciflow/inductor-periodic/162227 -> ciflow/inductor-periodic/162227
2025-09-07T07:36:04.3230442Z  * [new tag]                 ciflow/inductor-periodic/162323 -> ciflow/inductor-periodic/162323
2025-09-07T07:36:04.3231488Z  * [new tag]                 ciflow/inductor-rocm/154170 -> ciflow/inductor-rocm/154170
2025-09-07T07:36:04.3232471Z  * [new tag]                 ciflow/inductor-rocm/159146 -> ciflow/inductor-rocm/159146
2025-09-07T07:36:04.3233172Z  * [new tag]                 ciflow/inductor-rocm/159158 -> ciflow/inductor-rocm/159158
2025-09-07T07:36:04.3234139Z  * [new tag]                 ciflow/inductor-rocm/161715 -> ciflow/inductor-rocm/161715
2025-09-07T07:36:04.3235064Z  * [new tag]                 ciflow/inductor-rocm/162053 -> ciflow/inductor-rocm/162053
2025-09-07T07:36:04.3236003Z  * [new tag]                 ciflow/inductor-rocm/162056 -> ciflow/inductor-rocm/162056
2025-09-07T07:36:04.3237041Z  * [new tag]                 ciflow/inductor/137400      -> ciflow/inductor/137400
2025-09-07T07:36:04.3237831Z  * [new tag]                 ciflow/inductor/148180      -> ciflow/inductor/148180
2025-09-07T07:36:04.3238538Z  * [new tag]                 ciflow/inductor/148328      -> ciflow/inductor/148328
2025-09-07T07:36:04.3239347Z  * [new tag]                 ciflow/inductor/148484      -> ciflow/inductor/148484
2025-09-07T07:36:04.3240177Z  * [new tag]                 ciflow/inductor/148492      -> ciflow/inductor/148492
2025-09-07T07:36:04.3240832Z  * [new tag]                 ciflow/inductor/152624      -> ciflow/inductor/152624
2025-09-07T07:36:04.3241670Z  * [new tag]                 ciflow/inductor/154694      -> ciflow/inductor/154694
2025-09-07T07:36:04.3242359Z  * [new tag]                 ciflow/inductor/156049      -> ciflow/inductor/156049
2025-09-07T07:36:04.3243201Z  * [new tag]                 ciflow/inductor/156592      -> ciflow/inductor/156592
2025-09-07T07:36:04.3244040Z  * [new tag]                 ciflow/inductor/157635      -> ciflow/inductor/157635
2025-09-07T07:36:04.3244767Z  * [new tag]                 ciflow/inductor/157685      -> ciflow/inductor/157685
2025-09-07T07:36:04.3245795Z  * [new tag]                 ciflow/inductor/157686      -> ciflow/inductor/157686
2025-09-07T07:36:04.3246839Z  * [new tag]                 ciflow/inductor/157689      -> ciflow/inductor/157689
2025-09-07T07:36:04.3247851Z  * [new tag]                 ciflow/inductor/157699      -> ciflow/inductor/157699
2025-09-07T07:36:04.3248788Z  * [new tag]                 ciflow/inductor/157743      -> ciflow/inductor/157743
2025-09-07T07:36:04.3249733Z  * [new tag]                 ciflow/inductor/157994      -> ciflow/inductor/157994
2025-09-07T07:36:04.3250557Z  * [new tag]                 ciflow/inductor/158091      -> ciflow/inductor/158091
2025-09-07T07:36:04.3251470Z  * [new tag]                 ciflow/inductor/158104      -> ciflow/inductor/158104
2025-09-07T07:36:04.3252404Z  * [new tag]                 ciflow/inductor/158404      -> ciflow/inductor/158404
2025-09-07T07:36:04.3253255Z  * [new tag]                 ciflow/inductor/158647      -> ciflow/inductor/158647
2025-09-07T07:36:04.3254253Z  * [new tag]                 ciflow/inductor/158932      -> ciflow/inductor/158932
2025-09-07T07:36:04.3255187Z  * [new tag]                 ciflow/inductor/159146      -> ciflow/inductor/159146
2025-09-07T07:36:04.3256014Z  * [new tag]                 ciflow/inductor/159158      -> ciflow/inductor/159158
2025-09-07T07:36:04.3256947Z  * [new tag]                 ciflow/inductor/159274      -> ciflow/inductor/159274
2025-09-07T07:36:04.3257801Z  * [new tag]                 ciflow/inductor/159664      -> ciflow/inductor/159664
2025-09-07T07:36:04.3258896Z  * [new tag]                 ciflow/inductor/159778      -> ciflow/inductor/159778
2025-09-07T07:36:04.3259720Z  * [new tag]                 ciflow/inductor/159835      -> ciflow/inductor/159835
2025-09-07T07:36:04.3260889Z  * [new tag]                 ciflow/inductor/159944      -> ciflow/inductor/159944
2025-09-07T07:36:04.3261907Z  * [new tag]                 ciflow/inductor/160161      -> ciflow/inductor/160161
2025-09-07T07:36:04.3262730Z  * [new tag]                 ciflow/inductor/160174      -> ciflow/inductor/160174
2025-09-07T07:36:04.3263681Z  * [new tag]                 ciflow/inductor/160323      -> ciflow/inductor/160323
2025-09-07T07:36:04.3264765Z  * [new tag]                 ciflow/inductor/160324      -> ciflow/inductor/160324
2025-09-07T07:36:04.3265721Z  * [new tag]                 ciflow/inductor/160325      -> ciflow/inductor/160325
2025-09-07T07:36:04.3266732Z  * [new tag]                 ciflow/inductor/160326      -> ciflow/inductor/160326
2025-09-07T07:36:04.3267572Z  * [new tag]                 ciflow/inductor/160327      -> ciflow/inductor/160327
2025-09-07T07:36:04.3268456Z  * [new tag]                 ciflow/inductor/160328      -> ciflow/inductor/160328
2025-09-07T07:36:04.3269397Z  * [new tag]                 ciflow/inductor/160329      -> ciflow/inductor/160329
2025-09-07T07:36:04.3270241Z  * [new tag]                 ciflow/inductor/160480      -> ciflow/inductor/160480
2025-09-07T07:36:04.3271215Z  * [new tag]                 ciflow/inductor/160532      -> ciflow/inductor/160532
2025-09-07T07:36:04.3272583Z  * [new tag]                 ciflow/inductor/160539      -> ciflow/inductor/160539
2025-09-07T07:36:04.3273450Z  * [new tag]                 ciflow/inductor/160580      -> ciflow/inductor/160580
2025-09-07T07:36:04.3274306Z  * [new tag]                 ciflow/inductor/160685      -> ciflow/inductor/160685
2025-09-07T07:36:04.3275123Z  * [new tag]                 ciflow/inductor/160686      -> ciflow/inductor/160686
2025-09-07T07:36:04.3275962Z  * [new tag]                 ciflow/inductor/160687      -> ciflow/inductor/160687
2025-09-07T07:36:04.3276803Z  * [new tag]                 ciflow/inductor/160688      -> ciflow/inductor/160688
2025-09-07T07:36:04.3277643Z  * [new tag]                 ciflow/inductor/160690      -> ciflow/inductor/160690
2025-09-07T07:36:04.3278465Z  * [new tag]                 ciflow/inductor/160706      -> ciflow/inductor/160706
2025-09-07T07:36:04.3279380Z  * [new tag]                 ciflow/inductor/160729      -> ciflow/inductor/160729
2025-09-07T07:36:04.3280220Z  * [new tag]                 ciflow/inductor/160798      -> ciflow/inductor/160798
2025-09-07T07:36:04.3281230Z  * [new tag]                 ciflow/inductor/160836      -> ciflow/inductor/160836
2025-09-07T07:36:04.3282068Z  * [new tag]                 ciflow/inductor/160843      -> ciflow/inductor/160843
2025-09-07T07:36:04.3283193Z  * [new tag]                 ciflow/inductor/160869      -> ciflow/inductor/160869
2025-09-07T07:36:04.3284012Z  * [new tag]                 ciflow/inductor/160920      -> ciflow/inductor/160920
2025-09-07T07:36:04.3284874Z  * [new tag]                 ciflow/inductor/160943      -> ciflow/inductor/160943
2025-09-07T07:36:04.3285704Z  * [new tag]                 ciflow/inductor/161092      -> ciflow/inductor/161092
2025-09-07T07:36:04.3287004Z  * [new tag]                 ciflow/inductor/161093      -> ciflow/inductor/161093
2025-09-07T07:36:04.3287946Z  * [new tag]                 ciflow/inductor/161109      -> ciflow/inductor/161109
2025-09-07T07:36:04.3288775Z  * [new tag]                 ciflow/inductor/161118      -> ciflow/inductor/161118
2025-09-07T07:36:04.3289737Z  * [new tag]                 ciflow/inductor/161178      -> ciflow/inductor/161178
2025-09-07T07:36:04.3290665Z  * [new tag]                 ciflow/inductor/161246      -> ciflow/inductor/161246
2025-09-07T07:36:04.3291491Z  * [new tag]                 ciflow/inductor/161349      -> ciflow/inductor/161349
2025-09-07T07:36:04.3292350Z  * [new tag]                 ciflow/inductor/161350      -> ciflow/inductor/161350
2025-09-07T07:36:04.3293178Z  * [new tag]                 ciflow/inductor/161351      -> ciflow/inductor/161351
2025-09-07T07:36:04.3294146Z  * [new tag]                 ciflow/inductor/161397      -> ciflow/inductor/161397
2025-09-07T07:36:04.3295131Z  * [new tag]                 ciflow/inductor/161404      -> ciflow/inductor/161404
2025-09-07T07:36:04.3295767Z  * [new tag]                 ciflow/inductor/161405      -> ciflow/inductor/161405
2025-09-07T07:36:04.3296693Z  * [new tag]                 ciflow/inductor/161406      -> ciflow/inductor/161406
2025-09-07T07:36:04.3297740Z  * [new tag]                 ciflow/inductor/161410      -> ciflow/inductor/161410
2025-09-07T07:36:04.3298627Z  * [new tag]                 ciflow/inductor/161414      -> ciflow/inductor/161414
2025-09-07T07:36:04.3299654Z  * [new tag]                 ciflow/inductor/161442      -> ciflow/inductor/161442
2025-09-07T07:36:04.3300499Z  * [new tag]                 ciflow/inductor/161458      -> ciflow/inductor/161458
2025-09-07T07:36:04.3301311Z  * [new tag]                 ciflow/inductor/161468      -> ciflow/inductor/161468
2025-09-07T07:36:04.3302171Z  * [new tag]                 ciflow/inductor/161469      -> ciflow/inductor/161469
2025-09-07T07:36:04.3303240Z  * [new tag]                 ciflow/inductor/161485      -> ciflow/inductor/161485
2025-09-07T07:36:04.3304224Z  * [new tag]                 ciflow/inductor/161499      -> ciflow/inductor/161499
2025-09-07T07:36:04.3305044Z  * [new tag]                 ciflow/inductor/161534      -> ciflow/inductor/161534
2025-09-07T07:36:04.3306017Z  * [new tag]                 ciflow/inductor/161595      -> ciflow/inductor/161595
2025-09-07T07:36:04.3306859Z  * [new tag]                 ciflow/inductor/161596      -> ciflow/inductor/161596
2025-09-07T07:36:04.3308099Z  * [new tag]                 ciflow/inductor/161630      -> ciflow/inductor/161630
2025-09-07T07:36:04.3308923Z  * [new tag]                 ciflow/inductor/161667      -> ciflow/inductor/161667
2025-09-07T07:36:04.3309742Z  * [new tag]                 ciflow/inductor/161670      -> ciflow/inductor/161670
2025-09-07T07:36:04.3310662Z  * [new tag]                 ciflow/inductor/161673      -> ciflow/inductor/161673
2025-09-07T07:36:04.3311472Z  * [new tag]                 ciflow/inductor/161674      -> ciflow/inductor/161674
2025-09-07T07:36:04.3312511Z  * [new tag]                 ciflow/inductor/161675      -> ciflow/inductor/161675
2025-09-07T07:36:04.3313084Z  * [new tag]                 ciflow/inductor/161693      -> ciflow/inductor/161693
2025-09-07T07:36:04.3313964Z  * [new tag]                 ciflow/inductor/161695      -> ciflow/inductor/161695
2025-09-07T07:36:04.3314790Z  * [new tag]                 ciflow/inductor/161715      -> ciflow/inductor/161715
2025-09-07T07:36:04.3315630Z  * [new tag]                 ciflow/inductor/161730      -> ciflow/inductor/161730
2025-09-07T07:36:04.3316470Z  * [new tag]                 ciflow/inductor/161732      -> ciflow/inductor/161732
2025-09-07T07:36:04.3317383Z  * [new tag]                 ciflow/inductor/161744      -> ciflow/inductor/161744
2025-09-07T07:36:04.3318242Z  * [new tag]                 ciflow/inductor/161746      -> ciflow/inductor/161746
2025-09-07T07:36:04.3319040Z  * [new tag]                 ciflow/inductor/161747      -> ciflow/inductor/161747
2025-09-07T07:36:04.3319918Z  * [new tag]                 ciflow/inductor/161819      -> ciflow/inductor/161819
2025-09-07T07:36:04.3320802Z  * [new tag]                 ciflow/inductor/161821      -> ciflow/inductor/161821
2025-09-07T07:36:04.3321513Z  * [new tag]                 ciflow/inductor/161828      -> ciflow/inductor/161828
2025-09-07T07:36:04.3322306Z  * [new tag]                 ciflow/inductor/161879      -> ciflow/inductor/161879
2025-09-07T07:36:04.3323148Z  * [new tag]                 ciflow/inductor/161880      -> ciflow/inductor/161880
2025-09-07T07:36:04.3323969Z  * [new tag]                 ciflow/inductor/161881      -> ciflow/inductor/161881
2025-09-07T07:36:04.3325045Z  * [new tag]                 ciflow/inductor/161907      -> ciflow/inductor/161907
2025-09-07T07:36:04.3325809Z  * [new tag]                 ciflow/inductor/161914      -> ciflow/inductor/161914
2025-09-07T07:36:04.3326768Z  * [new tag]                 ciflow/inductor/161924      -> ciflow/inductor/161924
2025-09-07T07:36:04.3327650Z  * [new tag]                 ciflow/inductor/161936      -> ciflow/inductor/161936
2025-09-07T07:36:04.3328501Z  * [new tag]                 ciflow/inductor/161938      -> ciflow/inductor/161938
2025-09-07T07:36:04.3329385Z  * [new tag]                 ciflow/inductor/161939      -> ciflow/inductor/161939
2025-09-07T07:36:04.3330252Z  * [new tag]                 ciflow/inductor/161940      -> ciflow/inductor/161940
2025-09-07T07:36:04.3331078Z  * [new tag]                 ciflow/inductor/161955      -> ciflow/inductor/161955
2025-09-07T07:36:04.3331913Z  * [new tag]                 ciflow/inductor/161957      -> ciflow/inductor/161957
2025-09-07T07:36:04.3332920Z  * [new tag]                 ciflow/inductor/161975      -> ciflow/inductor/161975
2025-09-07T07:36:04.3333662Z  * [new tag]                 ciflow/inductor/161977      -> ciflow/inductor/161977
2025-09-07T07:36:04.3334421Z  * [new tag]                 ciflow/inductor/161978      -> ciflow/inductor/161978
2025-09-07T07:36:04.3335248Z  * [new tag]                 ciflow/inductor/161979      -> ciflow/inductor/161979
2025-09-07T07:36:04.3336066Z  * [new tag]                 ciflow/inductor/161980      -> ciflow/inductor/161980
2025-09-07T07:36:04.3337035Z  * [new tag]                 ciflow/inductor/161988      -> ciflow/inductor/161988
2025-09-07T07:36:04.3337843Z  * [new tag]                 ciflow/inductor/161994      -> ciflow/inductor/161994
2025-09-07T07:36:04.3338691Z  * [new tag]                 ciflow/inductor/162013      -> ciflow/inductor/162013
2025-09-07T07:36:04.3339522Z  * [new tag]                 ciflow/inductor/162014      -> ciflow/inductor/162014
2025-09-07T07:36:04.3340369Z  * [new tag]                 ciflow/inductor/162017      -> ciflow/inductor/162017
2025-09-07T07:36:04.3341266Z  * [new tag]                 ciflow/inductor/162021      -> ciflow/inductor/162021
2025-09-07T07:36:04.3342094Z  * [new tag]                 ciflow/inductor/162023      -> ciflow/inductor/162023
2025-09-07T07:36:04.3343335Z  * [new tag]                 ciflow/inductor/162027      -> ciflow/inductor/162027
2025-09-07T07:36:04.3344180Z  * [new tag]                 ciflow/inductor/162029      -> ciflow/inductor/162029
2025-09-07T07:36:04.3344984Z  * [new tag]                 ciflow/inductor/162030      -> ciflow/inductor/162030
2025-09-07T07:36:04.3345867Z  * [new tag]                 ciflow/inductor/162031      -> ciflow/inductor/162031
2025-09-07T07:36:04.3346636Z  * [new tag]                 ciflow/inductor/162033      -> ciflow/inductor/162033
2025-09-07T07:36:04.3347691Z  * [new tag]                 ciflow/inductor/162052      -> ciflow/inductor/162052
2025-09-07T07:36:04.3348421Z  * [new tag]                 ciflow/inductor/162053      -> ciflow/inductor/162053
2025-09-07T07:36:04.3349606Z  * [new tag]                 ciflow/inductor/162056      -> ciflow/inductor/162056
2025-09-07T07:36:04.3350347Z  * [new tag]                 ciflow/inductor/162063      -> ciflow/inductor/162063
2025-09-07T07:36:04.3351163Z  * [new tag]                 ciflow/inductor/162066      -> ciflow/inductor/162066
2025-09-07T07:36:04.3351992Z  * [new tag]                 ciflow/inductor/162068      -> ciflow/inductor/162068
2025-09-07T07:36:04.3353158Z  * [new tag]                 ciflow/inductor/162081      -> ciflow/inductor/162081
2025-09-07T07:36:04.3353827Z  * [new tag]                 ciflow/inductor/162088      -> ciflow/inductor/162088
2025-09-07T07:36:04.3354657Z  * [new tag]                 ciflow/inductor/162089      -> ciflow/inductor/162089
2025-09-07T07:36:04.3355477Z  * [new tag]                 ciflow/inductor/162094      -> ciflow/inductor/162094
2025-09-07T07:36:04.3356445Z  * [new tag]                 ciflow/inductor/162098      -> ciflow/inductor/162098
2025-09-07T07:36:04.3357202Z  * [new tag]                 ciflow/inductor/162101      -> ciflow/inductor/162101
2025-09-07T07:36:04.3358063Z  * [new tag]                 ciflow/inductor/162102      -> ciflow/inductor/162102
2025-09-07T07:36:04.3358903Z  * [new tag]                 ciflow/inductor/162104      -> ciflow/inductor/162104
2025-09-07T07:36:04.3359735Z  * [new tag]                 ciflow/inductor/162106      -> ciflow/inductor/162106
2025-09-07T07:36:04.3360566Z  * [new tag]                 ciflow/inductor/162108      -> ciflow/inductor/162108
2025-09-07T07:36:04.3361400Z  * [new tag]                 ciflow/inductor/162126      -> ciflow/inductor/162126
2025-09-07T07:36:04.3362265Z  * [new tag]                 ciflow/inductor/162149      -> ciflow/inductor/162149
2025-09-07T07:36:04.3363079Z  * [new tag]                 ciflow/inductor/162164      -> ciflow/inductor/162164
2025-09-07T07:36:04.3363908Z  * [new tag]                 ciflow/inductor/162166      -> ciflow/inductor/162166
2025-09-07T07:36:04.3364750Z  * [new tag]                 ciflow/inductor/162169      -> ciflow/inductor/162169
2025-09-07T07:36:04.3365703Z  * [new tag]                 ciflow/inductor/162170      -> ciflow/inductor/162170
2025-09-07T07:36:04.3366471Z  * [new tag]                 ciflow/inductor/162171      -> ciflow/inductor/162171
2025-09-07T07:36:04.3367316Z  * [new tag]                 ciflow/inductor/162183      -> ciflow/inductor/162183
2025-09-07T07:36:04.3368125Z  * [new tag]                 ciflow/inductor/162189      -> ciflow/inductor/162189
2025-09-07T07:36:04.3368953Z  * [new tag]                 ciflow/inductor/162190      -> ciflow/inductor/162190
2025-09-07T07:36:04.3369820Z  * [new tag]                 ciflow/inductor/162191      -> ciflow/inductor/162191
2025-09-07T07:36:04.3370666Z  * [new tag]                 ciflow/inductor/162194      -> ciflow/inductor/162194
2025-09-07T07:36:04.3371775Z  * [new tag]                 ciflow/inductor/162200      -> ciflow/inductor/162200
2025-09-07T07:36:04.3372556Z  * [new tag]                 ciflow/inductor/162201      -> ciflow/inductor/162201
2025-09-07T07:36:04.3373380Z  * [new tag]                 ciflow/inductor/162208      -> ciflow/inductor/162208
2025-09-07T07:36:04.3374450Z  * [new tag]                 ciflow/inductor/162211      -> ciflow/inductor/162211
2025-09-07T07:36:04.3375277Z  * [new tag]                 ciflow/inductor/162216      -> ciflow/inductor/162216
2025-09-07T07:36:04.3376033Z  * [new tag]                 ciflow/inductor/162220      -> ciflow/inductor/162220
2025-09-07T07:36:04.3377064Z  * [new tag]                 ciflow/inductor/162222      -> ciflow/inductor/162222
2025-09-07T07:36:04.3377922Z  * [new tag]                 ciflow/inductor/162227      -> ciflow/inductor/162227
2025-09-07T07:36:04.3378751Z  * [new tag]                 ciflow/inductor/162238      -> ciflow/inductor/162238
2025-09-07T07:36:04.3379600Z  * [new tag]                 ciflow/inductor/162239      -> ciflow/inductor/162239
2025-09-07T07:36:04.3380429Z  * [new tag]                 ciflow/inductor/162240      -> ciflow/inductor/162240
2025-09-07T07:36:04.3381275Z  * [new tag]                 ciflow/inductor/162244      -> ciflow/inductor/162244
2025-09-07T07:36:04.3382104Z  * [new tag]                 ciflow/inductor/162245      -> ciflow/inductor/162245
2025-09-07T07:36:04.3382943Z  * [new tag]                 ciflow/inductor/162262      -> ciflow/inductor/162262
2025-09-07T07:36:04.3383778Z  * [new tag]                 ciflow/inductor/162275      -> ciflow/inductor/162275
2025-09-07T07:36:04.3384617Z  * [new tag]                 ciflow/inductor/162278      -> ciflow/inductor/162278
2025-09-07T07:36:04.3385456Z  * [new tag]                 ciflow/inductor/162284      -> ciflow/inductor/162284
2025-09-07T07:36:04.3386280Z  * [new tag]                 ciflow/inductor/162286      -> ciflow/inductor/162286
2025-09-07T07:36:04.3387099Z  * [new tag]                 ciflow/inductor/162288      -> ciflow/inductor/162288
2025-09-07T07:36:04.3388077Z  * [new tag]                 ciflow/inductor/162293      -> ciflow/inductor/162293
2025-09-07T07:36:04.3388746Z  * [new tag]                 ciflow/inductor/162294      -> ciflow/inductor/162294
2025-09-07T07:36:04.3389573Z  * [new tag]                 ciflow/inductor/162295      -> ciflow/inductor/162295
2025-09-07T07:36:04.3390405Z  * [new tag]                 ciflow/inductor/162296      -> ciflow/inductor/162296
2025-09-07T07:36:04.3391224Z  * [new tag]                 ciflow/inductor/162298      -> ciflow/inductor/162298
2025-09-07T07:36:04.3392287Z  * [new tag]                 ciflow/inductor/162307      -> ciflow/inductor/162307
2025-09-07T07:36:04.3393143Z  * [new tag]                 ciflow/inductor/162309      -> ciflow/inductor/162309
2025-09-07T07:36:04.3393975Z  * [new tag]                 ciflow/inductor/162311      -> ciflow/inductor/162311
2025-09-07T07:36:04.3394924Z  * [new tag]                 ciflow/inductor/162312      -> ciflow/inductor/162312
2025-09-07T07:36:04.3395681Z  * [new tag]                 ciflow/inductor/162315      -> ciflow/inductor/162315
2025-09-07T07:36:04.3396500Z  * [new tag]                 ciflow/inductor/162316      -> ciflow/inductor/162316
2025-09-07T07:36:04.3397342Z  * [new tag]                 ciflow/inductor/162318      -> ciflow/inductor/162318
2025-09-07T07:36:04.3398613Z  * [new tag]                 ciflow/inductor/162323      -> ciflow/inductor/162323
2025-09-07T07:36:04.3399381Z  * [new tag]                 ciflow/inductor/162341      -> ciflow/inductor/162341
2025-09-07T07:36:04.3400180Z  * [new tag]                 ciflow/inductor/162345      -> ciflow/inductor/162345
2025-09-07T07:36:04.3401341Z  * [new tag]                 ciflow/inductor/3b9a386     -> ciflow/inductor/3b9a386
2025-09-07T07:36:04.3402338Z  * [new tag]                 ciflow/inductor/3d4b92b     -> ciflow/inductor/3d4b92b
2025-09-07T07:36:04.3403367Z  * [new tag]                 ciflow/inductor/d224ac7     -> ciflow/inductor/d224ac7
2025-09-07T07:36:04.3404512Z  * [new tag]                 ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994
2025-09-07T07:36:04.3405254Z  * [new tag]                 ciflow/linux-aarch64/159737 -> ciflow/linux-aarch64/159737
2025-09-07T07:36:04.3406022Z  * [new tag]                 ciflow/linux-aarch64/160078 -> ciflow/linux-aarch64/160078
2025-09-07T07:36:04.3407077Z  * [new tag]                 ciflow/mps/157553           -> ciflow/mps/157553
2025-09-07T07:36:04.3407963Z  * [new tag]                 ciflow/mps/157635           -> ciflow/mps/157635
2025-09-07T07:36:04.3408722Z  * [new tag]                 ciflow/mps/161988           -> ciflow/mps/161988
2025-09-07T07:36:04.3409474Z  * [new tag]                 ciflow/mps/162108           -> ciflow/mps/162108
2025-09-07T07:36:04.3410268Z  * [new tag]                 ciflow/mps/162153           -> ciflow/mps/162153
2025-09-07T07:36:04.3411043Z  * [new tag]                 ciflow/mps/162281           -> ciflow/mps/162281
2025-09-07T07:36:04.3412143Z  * [new tag]                 ciflow/nightly/156049       -> ciflow/nightly/156049
2025-09-07T07:36:04.3412826Z  * [new tag]                 ciflow/nightly/158104       -> ciflow/nightly/158104
2025-09-07T07:36:04.3413922Z  * [new tag]                 ciflow/op-benchmark/157994  -> ciflow/op-benchmark/157994
2025-09-07T07:36:04.3415110Z  * [new tag]                 ciflow/periodic-rocm-mi300/161529 -> ciflow/periodic-rocm-mi300/161529
2025-09-07T07:36:04.3415835Z  * [new tag]                 ciflow/periodic-rocm-mi300/161715 -> ciflow/periodic-rocm-mi300/161715
2025-09-07T07:36:04.3417112Z  * [new tag]                 ciflow/periodic/054a2fd     -> ciflow/periodic/054a2fd
2025-09-07T07:36:04.3417917Z  * [new tag]                 ciflow/periodic/156703      -> ciflow/periodic/156703
2025-09-07T07:36:04.3418684Z  * [new tag]                 ciflow/periodic/161715      -> ciflow/periodic/161715
2025-09-07T07:36:04.3419466Z  * [new tag]                 ciflow/periodic/162021      -> ciflow/periodic/162021
2025-09-07T07:36:04.3420213Z  * [new tag]                 ciflow/periodic/162323      -> ciflow/periodic/162323
2025-09-07T07:36:04.3421265Z  * [new tag]                 ciflow/periodic/2a6d37d     -> ciflow/periodic/2a6d37d
2025-09-07T07:36:04.3422075Z  * [new tag]                 ciflow/periodic/317eeb8     -> ciflow/periodic/317eeb8
2025-09-07T07:36:04.3422976Z  * [new tag]                 ciflow/periodic/3c32        -> ciflow/periodic/3c32
2025-09-07T07:36:04.3424018Z  * [new tag]                 ciflow/periodic/3e98831     -> ciflow/periodic/3e98831
2025-09-07T07:36:04.3425068Z  * [new tag]                 ciflow/periodic/94512-point -> ciflow/periodic/94512-point
2025-09-07T07:36:04.3426218Z  * [new tag]                 ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519
2025-09-07T07:36:04.3427496Z  * [new tag]                 ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275
2025-09-07T07:36:04.3428559Z  * [new tag]                 ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761
2025-09-07T07:36:04.3429666Z  * [new tag]                 ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12
2025-09-07T07:36:04.3430766Z  * [new tag]                 ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0
2025-09-07T07:36:04.3431863Z  * [new tag]                 ciflow/periodic/sha-ec5b83  -> ciflow/periodic/sha-ec5b83
2025-09-07T07:36:04.3432877Z  * [new tag]                 ciflow/rocm-mi300/154170    -> ciflow/rocm-mi300/154170
2025-09-07T07:36:04.3433708Z  * [new tag]                 ciflow/rocm-mi300/158747    -> ciflow/rocm-mi300/158747
2025-09-07T07:36:04.3434474Z  * [new tag]                 ciflow/rocm-mi300/159146    -> ciflow/rocm-mi300/159146
2025-09-07T07:36:04.3435267Z  * [new tag]                 ciflow/rocm-mi300/159158    -> ciflow/rocm-mi300/159158
2025-09-07T07:36:04.3436015Z  * [new tag]                 ciflow/rocm-mi300/161715    -> ciflow/rocm-mi300/161715
2025-09-07T07:36:04.3436813Z  * [new tag]                 ciflow/rocm-mi300/161957    -> ciflow/rocm-mi300/161957
2025-09-07T07:36:04.3437773Z  * [new tag]                 ciflow/rocm-mi300/162053    -> ciflow/rocm-mi300/162053
2025-09-07T07:36:04.3438962Z  * [new tag]                 ciflow/rocm-mi300/162056    -> ciflow/rocm-mi300/162056
2025-09-07T07:36:04.3439805Z  * [new tag]                 ciflow/rocm-mi300/162112    -> ciflow/rocm-mi300/162112
2025-09-07T07:36:04.3440578Z  * [new tag]                 ciflow/rocm-mi300/162245    -> ciflow/rocm-mi300/162245
2025-09-07T07:36:04.3441422Z  * [new tag]                 ciflow/rocm-mi300/162278    -> ciflow/rocm-mi300/162278
2025-09-07T07:36:04.3442465Z  * [new tag]                 ciflow/rocm-mi300/162288    -> ciflow/rocm-mi300/162288
2025-09-07T07:36:04.3443512Z  * [new tag]                 ciflow/rocm-mi355/162053    -> ciflow/rocm-mi355/162053
2025-09-07T07:36:04.3444316Z  * [new tag]                 ciflow/rocm-mi355/162056    -> ciflow/rocm-mi355/162056
2025-09-07T07:36:04.3445379Z  * [new tag]                 ciflow/rocm/148492          -> ciflow/rocm/148492
2025-09-07T07:36:04.3446095Z  * [new tag]                 ciflow/rocm/154170          -> ciflow/rocm/154170
2025-09-07T07:36:04.3447107Z  * [new tag]                 ciflow/rocm/156491          -> ciflow/rocm/156491
2025-09-07T07:36:04.3447830Z  * [new tag]                 ciflow/rocm/156592          -> ciflow/rocm/156592
2025-09-07T07:36:04.3448592Z  * [new tag]                 ciflow/rocm/158747          -> ciflow/rocm/158747
2025-09-07T07:36:04.3449342Z  * [new tag]                 ciflow/rocm/159146          -> ciflow/rocm/159146
2025-09-07T07:36:04.3450366Z  * [new tag]                 ciflow/rocm/159158          -> ciflow/rocm/159158
2025-09-07T07:36:04.3451065Z  * [new tag]                 ciflow/rocm/161715          -> ciflow/rocm/161715
2025-09-07T07:36:04.3452033Z  * [new tag]                 ciflow/rocm/161972          -> ciflow/rocm/161972
2025-09-07T07:36:04.3452735Z  * [new tag]                 ciflow/rocm/162052          -> ciflow/rocm/162052
2025-09-07T07:36:04.3453536Z  * [new tag]                 ciflow/rocm/162053          -> ciflow/rocm/162053
2025-09-07T07:36:04.3454697Z  * [new tag]                 ciflow/rocm/162056          -> ciflow/rocm/162056
2025-09-07T07:36:04.3455754Z  * [new tag]                 ciflow/rocm/162112          -> ciflow/rocm/162112
2025-09-07T07:36:04.3456663Z  * [new tag]                 ciflow/rocm/162278          -> ciflow/rocm/162278
2025-09-07T07:36:04.3458075Z  * [new tag]                 ciflow/rocm/162288          -> ciflow/rocm/162288
2025-09-07T07:36:04.3458876Z  * [new tag]                 ciflow/rocm/162305          -> ciflow/rocm/162305
2025-09-07T07:36:04.3460090Z  * [new tag]                 ciflow/slow/01c7106         -> ciflow/slow/01c7106
2025-09-07T07:36:04.3460905Z  * [new tag]                 ciflow/slow/0577043         -> ciflow/slow/0577043
2025-09-07T07:36:04.3462277Z  * [new tag]                 ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym
2025-09-07T07:36:04.3462915Z  * [new tag]                 ciflow/slow/0e81104         -> ciflow/slow/0e81104
2025-09-07T07:36:04.3463713Z  * [new tag]                 ciflow/slow/161395          -> ciflow/slow/161395
2025-09-07T07:36:04.3464696Z  * [new tag]                 ciflow/slow/1732077         -> ciflow/slow/1732077
2025-09-07T07:36:04.3465707Z  * [new tag]                 ciflow/slow/187eb7c         -> ciflow/slow/187eb7c
2025-09-07T07:36:04.3466458Z  * [new tag]                 ciflow/slow/1faef89         -> ciflow/slow/1faef89
2025-09-07T07:36:04.3467695Z  * [new tag]                 ciflow/slow/3920ec1         -> ciflow/slow/3920ec1
2025-09-07T07:36:04.3468812Z  * [new tag]                 ciflow/slow/3b7c6b2         -> ciflow/slow/3b7c6b2
2025-09-07T07:36:04.3469808Z  * [new tag]                 ciflow/slow/59a3759         -> ciflow/slow/59a3759
2025-09-07T07:36:04.3470767Z  * [new tag]                 ciflow/slow/70ef0bb         -> ciflow/slow/70ef0bb
2025-09-07T07:36:04.3471705Z  * [new tag]                 ciflow/slow/788ff06         -> ciflow/slow/788ff06
2025-09-07T07:36:04.3473041Z  * [new tag]                 ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym
2025-09-07T07:36:04.3473721Z  * [new tag]                 ciflow/slow/9d85864         -> ciflow/slow/9d85864
2025-09-07T07:36:04.3474777Z  * [new tag]                 ciflow/slow/9ffad5b         -> ciflow/slow/9ffad5b
2025-09-07T07:36:04.3475802Z  * [new tag]                 ciflow/slow/a206e8b         -> ciflow/slow/a206e8b
2025-09-07T07:36:04.3477063Z  * [new tag]                 ciflow/slow/a837609         -> ciflow/slow/a837609
2025-09-07T07:36:04.3478059Z  * [new tag]                 ciflow/slow/af841f3         -> ciflow/slow/af841f3
2025-09-07T07:36:04.3479290Z  * [new tag]                 ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym
2025-09-07T07:36:04.3480095Z  * [new tag]                 ciflow/triton_binaries/162329 -> ciflow/triton_binaries/162329
2025-09-07T07:36:04.3481111Z  * [new tag]                 ciflow/trunk/113258         -> ciflow/trunk/113258
2025-09-07T07:36:04.3481824Z  * [new tag]                 ciflow/trunk/137400         -> ciflow/trunk/137400
2025-09-07T07:36:04.3482589Z  * [new tag]                 ciflow/trunk/148180         -> ciflow/trunk/148180
2025-09-07T07:36:04.3483376Z  * [new tag]                 ciflow/trunk/148328         -> ciflow/trunk/148328
2025-09-07T07:36:04.3484133Z  * [new tag]                 ciflow/trunk/148492         -> ciflow/trunk/148492
2025-09-07T07:36:04.3485335Z  * [new tag]                 ciflow/trunk/148919         -> ciflow/trunk/148919
2025-09-07T07:36:04.3486045Z  * [new tag]                 ciflow/trunk/152624         -> ciflow/trunk/152624
2025-09-07T07:36:04.3486826Z  * [new tag]                 ciflow/trunk/154170         -> ciflow/trunk/154170
2025-09-07T07:36:04.3487573Z  * [new tag]                 ciflow/trunk/154694         -> ciflow/trunk/154694
2025-09-07T07:36:04.3488339Z  * [new tag]                 ciflow/trunk/156049         -> ciflow/trunk/156049
2025-09-07T07:36:04.3489102Z  * [new tag]                 ciflow/trunk/156703         -> ciflow/trunk/156703
2025-09-07T07:36:04.3490202Z  * [new tag]                 ciflow/trunk/156711         -> ciflow/trunk/156711
2025-09-07T07:36:04.3491245Z  * [new tag]                 ciflow/trunk/157432         -> ciflow/trunk/157432
2025-09-07T07:36:04.3492254Z  * [new tag]                 ciflow/trunk/157685         -> ciflow/trunk/157685
2025-09-07T07:36:04.3493013Z  * [new tag]                 ciflow/trunk/157689         -> ciflow/trunk/157689
2025-09-07T07:36:04.3493857Z  * [new tag]                 ciflow/trunk/157699         -> ciflow/trunk/157699
2025-09-07T07:36:04.3494679Z  * [new tag]                 ciflow/trunk/157813         -> ciflow/trunk/157813
2025-09-07T07:36:04.3495490Z  * [new tag]                 ciflow/trunk/157994         -> ciflow/trunk/157994
2025-09-07T07:36:04.3496347Z  * [new tag]                 ciflow/trunk/158091         -> ciflow/trunk/158091
2025-09-07T07:36:04.3497175Z  * [new tag]                 ciflow/trunk/158104         -> ciflow/trunk/158104
2025-09-07T07:36:04.3498500Z  * [new tag]                 ciflow/trunk/158404         -> ciflow/trunk/158404
2025-09-07T07:36:04.3499316Z  * [new tag]                 ciflow/trunk/158647         -> ciflow/trunk/158647
2025-09-07T07:36:04.3500458Z  * [new tag]                 ciflow/trunk/158846         -> ciflow/trunk/158846
2025-09-07T07:36:04.3501217Z  * [new tag]                 ciflow/trunk/159158         -> ciflow/trunk/159158
2025-09-07T07:36:04.3502202Z  * [new tag]                 ciflow/trunk/159682         -> ciflow/trunk/159682
2025-09-07T07:36:04.3502959Z  * [new tag]                 ciflow/trunk/159835         -> ciflow/trunk/159835
2025-09-07T07:36:04.3504064Z  * [new tag]                 ciflow/trunk/160161         -> ciflow/trunk/160161
2025-09-07T07:36:04.3504837Z  * [new tag]                 ciflow/trunk/160236         -> ciflow/trunk/160236
2025-09-07T07:36:04.3505655Z  * [new tag]                 ciflow/trunk/160329         -> ciflow/trunk/160329
2025-09-07T07:36:04.3506469Z  * [new tag]                 ciflow/trunk/160480         -> ciflow/trunk/160480
2025-09-07T07:36:04.3507296Z  * [new tag]                 ciflow/trunk/160532         -> ciflow/trunk/160532
2025-09-07T07:36:04.3508088Z  * [new tag]                 ciflow/trunk/160836         -> ciflow/trunk/160836
2025-09-07T07:36:04.3509034Z  * [new tag]                 ciflow/trunk/160843         -> ciflow/trunk/160843
2025-09-07T07:36:04.3509749Z  * [new tag]                 ciflow/trunk/160869         -> ciflow/trunk/160869
2025-09-07T07:36:04.3510796Z  * [new tag]                 ciflow/trunk/160940         -> ciflow/trunk/160940
2025-09-07T07:36:04.3511572Z  * [new tag]                 ciflow/trunk/160943         -> ciflow/trunk/160943
2025-09-07T07:36:04.3512652Z  * [new tag]                 ciflow/trunk/160953         -> ciflow/trunk/160953
2025-09-07T07:36:04.3513635Z  * [new tag]                 ciflow/trunk/161035         -> ciflow/trunk/161035
2025-09-07T07:36:04.3514459Z  * [new tag]                 ciflow/trunk/161178         -> ciflow/trunk/161178
2025-09-07T07:36:04.3515274Z  * [new tag]                 ciflow/trunk/161349         -> ciflow/trunk/161349
2025-09-07T07:36:04.3516105Z  * [new tag]                 ciflow/trunk/161350         -> ciflow/trunk/161350
2025-09-07T07:36:04.3516924Z  * [new tag]                 ciflow/trunk/161351         -> ciflow/trunk/161351
2025-09-07T07:36:04.3517732Z  * [new tag]                 ciflow/trunk/161395         -> ciflow/trunk/161395
2025-09-07T07:36:04.3518534Z  * [new tag]                 ciflow/trunk/161405         -> ciflow/trunk/161405
2025-09-07T07:36:04.3519348Z  * [new tag]                 ciflow/trunk/161406         -> ciflow/trunk/161406
2025-09-07T07:36:04.3520167Z  * [new tag]                 ciflow/trunk/161410         -> ciflow/trunk/161410
2025-09-07T07:36:04.3521001Z  * [new tag]                 ciflow/trunk/161468         -> ciflow/trunk/161468
2025-09-07T07:36:04.3521824Z  * [new tag]                 ciflow/trunk/161499         -> ciflow/trunk/161499
2025-09-07T07:36:04.3522957Z  * [new tag]                 ciflow/trunk/161527         -> ciflow/trunk/161527
2025-09-07T07:36:04.3523786Z  * [new tag]                 ciflow/trunk/161534         -> ciflow/trunk/161534
2025-09-07T07:36:04.3524563Z  * [new tag]                 ciflow/trunk/161591         -> ciflow/trunk/161591
2025-09-07T07:36:04.3525368Z  * [new tag]                 ciflow/trunk/161595         -> ciflow/trunk/161595
2025-09-07T07:36:04.3526195Z  * [new tag]                 ciflow/trunk/161596         -> ciflow/trunk/161596
2025-09-07T07:36:04.3527017Z  * [new tag]                 ciflow/trunk/161633         -> ciflow/trunk/161633
2025-09-07T07:36:04.3527965Z  * [new tag]                 ciflow/trunk/161634         -> ciflow/trunk/161634
2025-09-07T07:36:04.3528925Z  * [new tag]                 ciflow/trunk/161635         -> ciflow/trunk/161635
2025-09-07T07:36:04.3529703Z  * [new tag]                 ciflow/trunk/161667         -> ciflow/trunk/161667
2025-09-07T07:36:04.3530517Z  * [new tag]                 ciflow/trunk/161670         -> ciflow/trunk/161670
2025-09-07T07:36:04.3531346Z  * [new tag]                 ciflow/trunk/161692         -> ciflow/trunk/161692
2025-09-07T07:36:04.3532176Z  * [new tag]                 ciflow/trunk/161693         -> ciflow/trunk/161693
2025-09-07T07:36:04.3533110Z  * [new tag]                 ciflow/trunk/161695         -> ciflow/trunk/161695
2025-09-07T07:36:04.3533875Z  * [new tag]                 ciflow/trunk/161730         -> ciflow/trunk/161730
2025-09-07T07:36:04.3534687Z  * [new tag]                 ciflow/trunk/161744         -> ciflow/trunk/161744
2025-09-07T07:36:04.3535497Z  * [new tag]                 ciflow/trunk/161749         -> ciflow/trunk/161749
2025-09-07T07:36:04.3536326Z  * [new tag]                 ciflow/trunk/161881         -> ciflow/trunk/161881
2025-09-07T07:36:04.3537137Z  * [new tag]                 ciflow/trunk/161924         -> ciflow/trunk/161924
2025-09-07T07:36:04.3538228Z  * [new tag]                 ciflow/trunk/161926         -> ciflow/trunk/161926
2025-09-07T07:36:04.3539010Z  * [new tag]                 ciflow/trunk/161936         -> ciflow/trunk/161936
2025-09-07T07:36:04.3539837Z  * [new tag]                 ciflow/trunk/161952         -> ciflow/trunk/161952
2025-09-07T07:36:04.3540770Z  * [new tag]                 ciflow/trunk/161955         -> ciflow/trunk/161955
2025-09-07T07:36:04.3541487Z  * [new tag]                 ciflow/trunk/161957         -> ciflow/trunk/161957
2025-09-07T07:36:04.3542298Z  * [new tag]                 ciflow/trunk/161959         -> ciflow/trunk/161959
2025-09-07T07:36:04.3543109Z  * [new tag]                 ciflow/trunk/161977         -> ciflow/trunk/161977
2025-09-07T07:36:04.3543920Z  * [new tag]                 ciflow/trunk/161988         -> ciflow/trunk/161988
2025-09-07T07:36:04.3544756Z  * [new tag]                 ciflow/trunk/161994         -> ciflow/trunk/161994
2025-09-07T07:36:04.3545761Z  * [new tag]                 ciflow/trunk/162007         -> ciflow/trunk/162007
2025-09-07T07:36:04.3546519Z  * [new tag]                 ciflow/trunk/162013         -> ciflow/trunk/162013
2025-09-07T07:36:04.3547342Z  * [new tag]                 ciflow/trunk/162017         -> ciflow/trunk/162017
2025-09-07T07:36:04.3548176Z  * [new tag]                 ciflow/trunk/162021         -> ciflow/trunk/162021
2025-09-07T07:36:04.3549008Z  * [new tag]                 ciflow/trunk/162022         -> ciflow/trunk/162022
2025-09-07T07:36:04.3549850Z  * [new tag]                 ciflow/trunk/162040         -> ciflow/trunk/162040
2025-09-07T07:36:04.3550677Z  * [new tag]                 ciflow/trunk/162041         -> ciflow/trunk/162041
2025-09-07T07:36:04.3551679Z  * [new tag]                 ciflow/trunk/162062         -> ciflow/trunk/162062
2025-09-07T07:36:04.3552457Z  * [new tag]                 ciflow/trunk/162066         -> ciflow/trunk/162066
2025-09-07T07:36:04.3553701Z  * [new tag]                 ciflow/trunk/162089         -> ciflow/trunk/162089
2025-09-07T07:36:04.3554486Z  * [new tag]                 ciflow/trunk/162099         -> ciflow/trunk/162099
2025-09-07T07:36:04.3555452Z  * [new tag]                 ciflow/trunk/162104         -> ciflow/trunk/162104
2025-09-07T07:36:04.3556231Z  * [new tag]                 ciflow/trunk/162106         -> ciflow/trunk/162106
2025-09-07T07:36:04.3557056Z  * [new tag]                 ciflow/trunk/162112         -> ciflow/trunk/162112
2025-09-07T07:36:04.3557889Z  * [new tag]                 ciflow/trunk/162119         -> ciflow/trunk/162119
2025-09-07T07:36:04.3558734Z  * [new tag]                 ciflow/trunk/162142         -> ciflow/trunk/162142
2025-09-07T07:36:04.3559552Z  * [new tag]                 ciflow/trunk/162169         -> ciflow/trunk/162169
2025-09-07T07:36:04.3560363Z  * [new tag]                 ciflow/trunk/162183         -> ciflow/trunk/162183
2025-09-07T07:36:04.3561174Z  * [new tag]                 ciflow/trunk/162190         -> ciflow/trunk/162190
2025-09-07T07:36:04.3561989Z  * [new tag]                 ciflow/trunk/162194         -> ciflow/trunk/162194
2025-09-07T07:36:04.3562808Z  * [new tag]                 ciflow/trunk/162200         -> ciflow/trunk/162200
2025-09-07T07:36:04.3563637Z  * [new tag]                 ciflow/trunk/162206         -> ciflow/trunk/162206
2025-09-07T07:36:04.3564478Z  * [new tag]                 ciflow/trunk/162208         -> ciflow/trunk/162208
2025-09-07T07:36:04.3565482Z  * [new tag]                 ciflow/trunk/162222         -> ciflow/trunk/162222
2025-09-07T07:36:04.3566249Z  * [new tag]                 ciflow/trunk/162238         -> ciflow/trunk/162238
2025-09-07T07:36:04.3567064Z  * [new tag]                 ciflow/trunk/162244         -> ciflow/trunk/162244
2025-09-07T07:36:04.3568211Z  * [new tag]                 ciflow/trunk/162267         -> ciflow/trunk/162267
2025-09-07T07:36:04.3569291Z  * [new tag]                 ciflow/trunk/162269         -> ciflow/trunk/162269
2025-09-07T07:36:04.3569932Z  * [new tag]                 ciflow/trunk/162278         -> ciflow/trunk/162278
2025-09-07T07:36:04.3570990Z  * [new tag]                 ciflow/trunk/162286         -> ciflow/trunk/162286
2025-09-07T07:36:04.3571752Z  * [new tag]                 ciflow/trunk/162288         -> ciflow/trunk/162288
2025-09-07T07:36:04.3572598Z  * [new tag]                 ciflow/trunk/162293         -> ciflow/trunk/162293
2025-09-07T07:36:04.3573491Z  * [new tag]                 ciflow/trunk/162310         -> ciflow/trunk/162310
2025-09-07T07:36:04.3574243Z  * [new tag]                 ciflow/trunk/162311         -> ciflow/trunk/162311
2025-09-07T07:36:04.3575065Z  * [new tag]                 ciflow/trunk/162315         -> ciflow/trunk/162315
2025-09-07T07:36:04.3575882Z  * [new tag]                 ciflow/trunk/162325         -> ciflow/trunk/162325
2025-09-07T07:36:04.3576904Z  * [new tag]                 ciflow/trunk/162328         -> ciflow/trunk/162328
2025-09-07T07:36:04.3577732Z  * [new tag]                 ciflow/trunk/162329         -> ciflow/trunk/162329
2025-09-07T07:36:04.3579075Z  * [new tag]                 ciflow/unstable/123         -> ciflow/unstable/123
2025-09-07T07:36:04.3580111Z  * [new tag]                 ciflow/vllm/162292          -> ciflow/vllm/162292
2025-09-07T07:36:04.3593375Z  * [new tag]                 ciflow/win-arm64/156049     -> ciflow/win-arm64/156049
2025-09-07T07:36:04.3593710Z  * [new tag]                 ciflow/win-arm64/158104     -> ciflow/win-arm64/158104
2025-09-07T07:36:04.3594017Z  * [new tag]                 ciflow/xpu/157699           -> ciflow/xpu/157699
2025-09-07T07:36:04.3594294Z  * [new tag]                 ciflow/xpu/157994           -> ciflow/xpu/157994
2025-09-07T07:36:04.3594543Z  * [new tag]                 ciflow/xpu/159459           -> ciflow/xpu/159459
2025-09-07T07:36:04.3594799Z  * [new tag]                 ciflow/xpu/159718           -> ciflow/xpu/159718
2025-09-07T07:36:04.3595053Z  * [new tag]                 ciflow/xpu/159944           -> ciflow/xpu/159944
2025-09-07T07:36:04.3595308Z  * [new tag]                 ciflow/xpu/160867           -> ciflow/xpu/160867
2025-09-07T07:36:04.3595555Z  * [new tag]                 ciflow/xpu/160938           -> ciflow/xpu/160938
2025-09-07T07:36:04.3595806Z  * [new tag]                 ciflow/xpu/160940           -> ciflow/xpu/160940
2025-09-07T07:36:04.3596171Z  * [new tag]                 ciflow/xpu/160953           -> ciflow/xpu/160953
2025-09-07T07:36:04.3596429Z  * [new tag]                 ciflow/xpu/161045           -> ciflow/xpu/161045
2025-09-07T07:36:04.3596683Z  * [new tag]                 ciflow/xpu/161058           -> ciflow/xpu/161058
2025-09-07T07:36:04.3596931Z  * [new tag]                 ciflow/xpu/161246           -> ciflow/xpu/161246
2025-09-07T07:36:04.3597182Z  * [new tag]                 ciflow/xpu/161397           -> ciflow/xpu/161397
2025-09-07T07:36:04.3597443Z  * [new tag]                 ciflow/xpu/161485           -> ciflow/xpu/161485
2025-09-07T07:36:04.3597707Z  * [new tag]                 ciflow/xpu/161988           -> ciflow/xpu/161988
2025-09-07T07:36:04.3597965Z  * [new tag]                 ciflow/xpu/162062           -> ciflow/xpu/162062
2025-09-07T07:36:04.3598216Z  * [new tag]                 cslpull75                   -> cslpull75
2025-09-07T07:36:04.3598626Z  * [new tag]                 cslpull76                   -> cslpull76
2025-09-07T07:36:04.3599423Z  * [new tag]                 cslpull77                   -> cslpull77
2025-09-07T07:36:04.3600255Z  * [new tag]                 cslpull78                   -> cslpull78
2025-09-07T07:36:04.3601409Z  * [new tag]                 cslpull79                   -> cslpull79
2025-09-07T07:36:04.3602566Z  * [new tag]                 cslpull80                   -> cslpull80
2025-09-07T07:36:04.3606305Z  * [new tag]                 cslpull81                   -> cslpull81
2025-09-07T07:36:04.3607311Z  * [new tag]                 cslpull82                   -> cslpull82
2025-09-07T07:36:04.3608149Z  * [new tag]                 cslpull83                   -> cslpull83
2025-09-07T07:36:04.3609126Z  * [new tag]                 cslpull84                   -> cslpull84
2025-09-07T07:36:04.3610133Z  * [new tag]                 cslpull85                   -> cslpull85
2025-09-07T07:36:04.3610896Z  * [new tag]                 cslpull86                   -> cslpull86
2025-09-07T07:36:04.3611833Z  * [new tag]                 cslpull87                   -> cslpull87
2025-09-07T07:36:04.3612975Z  * [new tag]                 cslpull88                   -> cslpull88
2025-09-07T07:36:04.3613784Z  * [new tag]                 cslpull89                   -> cslpull89
2025-09-07T07:36:04.3614574Z  * [new tag]                 cslpull90                   -> cslpull90
2025-09-07T07:36:04.3615788Z  * [new tag]                 cslpull91                   -> cslpull91
2025-09-07T07:36:04.3616604Z  * [new tag]                 cslpull92                   -> cslpull92
2025-09-07T07:36:04.3617601Z  * [new tag]                 flight_5                    -> flight_5
2025-09-07T07:36:04.3618684Z  * [new tag]                 flight_5.1                  -> flight_5.1
2025-09-07T07:36:04.3619656Z  * [new tag]                 flight_5.2                  -> flight_5.2
2025-09-07T07:36:04.3620840Z  * [new tag]                 flight_5.3                  -> flight_5.3
2025-09-07T07:36:04.3621570Z  * [new tag]                 forpull1                    -> forpull1
2025-09-07T07:36:04.3622774Z  * [new tag]                 malfet/tag-2ef5611          -> malfet/tag-2ef5611
2025-09-07T07:36:04.3623610Z  * [new tag]                 malfet/tag-317b1a0          -> malfet/tag-317b1a0
2025-09-07T07:36:04.3624575Z  * [new tag]                 malfet/tag-ec6f767          -> malfet/tag-ec6f767
2025-09-07T07:36:04.3625557Z  * [new tag]                 nightly-binary              -> nightly-binary
2025-09-07T07:36:04.3626268Z  * [new tag]                 sqzhang_flight4_plus        -> sqzhang_flight4_plus
2025-09-07T07:36:04.3627305Z  * [new tag]                 sqzhang_flight_3            -> sqzhang_flight_3
2025-09-07T07:36:04.3628619Z  * [new tag]                 trunk/00636e0171e7e733628c408084805442270cf608 -> trunk/00636e0171e7e733628c408084805442270cf608
2025-09-07T07:36:04.3629419Z  * [new tag]                 trunk/019fed39aa6b2dd8c69347378d53423e5efae8d4 -> trunk/019fed39aa6b2dd8c69347378d53423e5efae8d4
2025-09-07T07:36:04.3630612Z  * [new tag]                 trunk/01ab325cc2e0dc221af4d710974e1b9175066544 -> trunk/01ab325cc2e0dc221af4d710974e1b9175066544
2025-09-07T07:36:04.3631726Z  * [new tag]                 trunk/01edcd4df8bf0c7b4cc2d3ec868bd2059eeea83b -> trunk/01edcd4df8bf0c7b4cc2d3ec868bd2059eeea83b
2025-09-07T07:36:04.3632725Z  * [new tag]                 trunk/040d00af048967dde7938d358d7f5988cbd18388 -> trunk/040d00af048967dde7938d358d7f5988cbd18388
2025-09-07T07:36:04.3633716Z  * [new tag]                 trunk/0447f2d99b4351b2ff129dce6eebb371024f73e5 -> trunk/0447f2d99b4351b2ff129dce6eebb371024f73e5
2025-09-07T07:36:04.3634714Z  * [new tag]                 trunk/047603d35bdc70046216384838d6340feab79bf4 -> trunk/047603d35bdc70046216384838d6340feab79bf4
2025-09-07T07:36:04.3635665Z  * [new tag]                 trunk/06da7c0730b3764f178ec3a90dedf4ffa4202d81 -> trunk/06da7c0730b3764f178ec3a90dedf4ffa4202d81
2025-09-07T07:36:04.3636738Z  * [new tag]                 trunk/081cab045472ce045634548cc6c14a4870641e23 -> trunk/081cab045472ce045634548cc6c14a4870641e23
2025-09-07T07:36:04.3637682Z  * [new tag]                 trunk/09587daf8c9f21f5340f73921ce5f23d1a4a4572 -> trunk/09587daf8c9f21f5340f73921ce5f23d1a4a4572
2025-09-07T07:36:04.3638596Z  * [new tag]                 trunk/09be1890d72cc34fc946965dc4a27736bf0ca8c6 -> trunk/09be1890d72cc34fc946965dc4a27736bf0ca8c6
2025-09-07T07:36:04.3639544Z  * [new tag]                 trunk/09d2f1b6315d6d416fbf452793d65795863ebc66 -> trunk/09d2f1b6315d6d416fbf452793d65795863ebc66
2025-09-07T07:36:04.3640435Z  * [new tag]                 trunk/0af70e2353e1dcda83175fd4834ecb7b63e009e0 -> trunk/0af70e2353e1dcda83175fd4834ecb7b63e009e0
2025-09-07T07:36:04.3641951Z  * [new tag]                 trunk/0c0e056a9e20c17271a6144dd32c0c7e3ba26736 -> trunk/0c0e056a9e20c17271a6144dd32c0c7e3ba26736
2025-09-07T07:36:04.3642921Z  * [new tag]                 trunk/0cd6c56bdfa9178ff61be82ce3b178926ddb64a9 -> trunk/0cd6c56bdfa9178ff61be82ce3b178926ddb64a9
2025-09-07T07:36:04.3643840Z  * [new tag]                 trunk/0d421ace32c1605ee8e452ee1eeb03bd243dd96c -> trunk/0d421ace32c1605ee8e452ee1eeb03bd243dd96c
2025-09-07T07:36:04.3644986Z  * [new tag]                 trunk/0d71a9dd5b4b6d1dde58d91c9b71d96bc6a6a171 -> trunk/0d71a9dd5b4b6d1dde58d91c9b71d96bc6a6a171
2025-09-07T07:36:04.3645835Z  * [new tag]                 trunk/0d84ff3b78f55492d3d4708458c92d776274939e -> trunk/0d84ff3b78f55492d3d4708458c92d776274939e
2025-09-07T07:36:04.3646758Z  * [new tag]                 trunk/0f45aaf4414048b17d720d0915ce221a8de8ec63 -> trunk/0f45aaf4414048b17d720d0915ce221a8de8ec63
2025-09-07T07:36:04.3647692Z  * [new tag]                 trunk/0ff8eabf1387de5acd6712a03bda61f1a3dfa27f -> trunk/0ff8eabf1387de5acd6712a03bda61f1a3dfa27f
2025-09-07T07:36:04.3648637Z  * [new tag]                 trunk/104f2680e03d13a4765ca69f905d8f16fc0c822f -> trunk/104f2680e03d13a4765ca69f905d8f16fc0c822f
2025-09-07T07:36:04.3649600Z  * [new tag]                 trunk/12814701555d3e41dfcdf8f9273af5821e322df0 -> trunk/12814701555d3e41dfcdf8f9273af5821e322df0
2025-09-07T07:36:04.3650574Z  * [new tag]                 trunk/13b65196db422bdb394cb482e208c61ed448898c -> trunk/13b65196db422bdb394cb482e208c61ed448898c
2025-09-07T07:36:04.3651482Z  * [new tag]                 trunk/13d66e2a66eceed14b8a8f5a971087df4f688a46 -> trunk/13d66e2a66eceed14b8a8f5a971087df4f688a46
2025-09-07T07:36:04.3652398Z  * [new tag]                 trunk/145a3a7bda15e3963a33eb1b54bba5d4a270b225 -> trunk/145a3a7bda15e3963a33eb1b54bba5d4a270b225
2025-09-07T07:36:04.3653349Z  * [new tag]                 trunk/146371483318e17929daefd37c8e459d9d6d47bb -> trunk/146371483318e17929daefd37c8e459d9d6d47bb
2025-09-07T07:36:04.3654334Z  * [new tag]                 trunk/15c77a8cfd341e74fd124b077492ef2bfa51b339 -> trunk/15c77a8cfd341e74fd124b077492ef2bfa51b339
2025-09-07T07:36:04.3655274Z  * [new tag]                 trunk/17fa8eec4a1e32939ab4d364ee6e75487a79b654 -> trunk/17fa8eec4a1e32939ab4d364ee6e75487a79b654
2025-09-07T07:36:04.3656689Z  * [new tag]                 trunk/190c391a28845a14df26abb228d26aa813efb20c -> trunk/190c391a28845a14df26abb228d26aa813efb20c
2025-09-07T07:36:04.3657665Z  * [new tag]                 trunk/1a588ace4667bde1331fbd8ed957157dca5cee68 -> trunk/1a588ace4667bde1331fbd8ed957157dca5cee68
2025-09-07T07:36:04.3658666Z  * [new tag]                 trunk/1aa7476885e8f6e7b0ec3a5b6383aad9d3f343e7 -> trunk/1aa7476885e8f6e7b0ec3a5b6383aad9d3f343e7
2025-09-07T07:36:04.3659474Z  * [new tag]                 trunk/1aeb421c342c9e9607842f4c87cb46e8e816ee53 -> trunk/1aeb421c342c9e9607842f4c87cb46e8e816ee53
2025-09-07T07:36:04.3660412Z  * [new tag]                 trunk/1c1b28d5b6a942fafe23b2f09302d93c25226d4a -> trunk/1c1b28d5b6a942fafe23b2f09302d93c25226d4a
2025-09-07T07:36:04.3661320Z  * [new tag]                 trunk/1ebd70d0c0d562d3be9abdee2a21906584af7d99 -> trunk/1ebd70d0c0d562d3be9abdee2a21906584af7d99
2025-09-07T07:36:04.3662265Z  * [new tag]                 trunk/1ec2c15914da4ef7bd926ed9aebc8671c75fe965 -> trunk/1ec2c15914da4ef7bd926ed9aebc8671c75fe965
2025-09-07T07:36:04.3663203Z  * [new tag]                 trunk/1f51056bd64e73d1aa81321bc3c098575b1bc78a -> trunk/1f51056bd64e73d1aa81321bc3c098575b1bc78a
2025-09-07T07:36:04.3664199Z  * [new tag]                 trunk/1f820de639c75a1562d3fb03f160439f853ae07b -> trunk/1f820de639c75a1562d3fb03f160439f853ae07b
2025-09-07T07:36:04.3665258Z  * [new tag]                 trunk/204697f0e695d82894c5010fbec664c4391f90cc -> trunk/204697f0e695d82894c5010fbec664c4391f90cc
2025-09-07T07:36:04.3666153Z  * [new tag]                 trunk/20629b1619fe636227d01fc85ba221daa7185a05 -> trunk/20629b1619fe636227d01fc85ba221daa7185a05
2025-09-07T07:36:04.3667088Z  * [new tag]                 trunk/20b47acef845e9c4f71da9429a396d293f50ebe7 -> trunk/20b47acef845e9c4f71da9429a396d293f50ebe7
2025-09-07T07:36:04.3668026Z  * [new tag]                 trunk/20bfb2539d7c5250379648eda35f80b8a7d642dd -> trunk/20bfb2539d7c5250379648eda35f80b8a7d642dd
2025-09-07T07:36:04.3669028Z  * [new tag]                 trunk/21fae99c180d17def562797ea0fb154d8fdf88e3 -> trunk/21fae99c180d17def562797ea0fb154d8fdf88e3
2025-09-07T07:36:04.3670235Z  * [new tag]                 trunk/248355faf53f9f7ba2fd0a367d59600c6d991e7f -> trunk/248355faf53f9f7ba2fd0a367d59600c6d991e7f
2025-09-07T07:36:04.3671105Z  * [new tag]                 trunk/25f4aaed9ec26f39c13862323ff8582006473d23 -> trunk/25f4aaed9ec26f39c13862323ff8582006473d23
2025-09-07T07:36:04.3672073Z  * [new tag]                 trunk/261a84a1764412f8e659c956e3f81997ec3de9d5 -> trunk/261a84a1764412f8e659c956e3f81997ec3de9d5
2025-09-07T07:36:04.3673117Z  * [new tag]                 trunk/28f4ab0737937858730f29f5c4e601e109cf9d5f -> trunk/28f4ab0737937858730f29f5c4e601e109cf9d5f
2025-09-07T07:36:04.3674140Z  * [new tag]                 trunk/291cd11f2d5df6f48d348cce0e4e762f274f4dc4 -> trunk/291cd11f2d5df6f48d348cce0e4e762f274f4dc4
2025-09-07T07:36:04.3675080Z  * [new tag]                 trunk/29280864d941e6108ab57f7298f520c0cf9696e9 -> trunk/29280864d941e6108ab57f7298f520c0cf9696e9
2025-09-07T07:36:04.3676118Z  * [new tag]                 trunk/2a45837e98c63cae9d1a2e2133a727b829e549d5 -> trunk/2a45837e98c63cae9d1a2e2133a727b829e549d5
2025-09-07T07:36:04.3677177Z  * [new tag]                 trunk/2a5c0785e2f975697fd7bdf1411de6e03dcaa1ef -> trunk/2a5c0785e2f975697fd7bdf1411de6e03dcaa1ef
2025-09-07T07:36:04.3678154Z  * [new tag]                 trunk/2b8a83901c58a0858ea9e4ce00055f48e6ed164c -> trunk/2b8a83901c58a0858ea9e4ce00055f48e6ed164c
2025-09-07T07:36:04.3679060Z  * [new tag]                 trunk/2ba65472dd54488a86a50326ea990195fc6732d6 -> trunk/2ba65472dd54488a86a50326ea990195fc6732d6
2025-09-07T07:36:04.3680001Z  * [new tag]                 trunk/2c03f0acc53ed13fe8ebfe809129f25996e009a0 -> trunk/2c03f0acc53ed13fe8ebfe809129f25996e009a0
2025-09-07T07:36:04.3680905Z  * [new tag]                 trunk/2dd529df0092799f68ee7afcf52338276906706a -> trunk/2dd529df0092799f68ee7afcf52338276906706a
2025-09-07T07:36:04.3681868Z  * [new tag]                 trunk/2f6b4b1ad3f82bb3bd984f6e65744ea339ffb8b5 -> trunk/2f6b4b1ad3f82bb3bd984f6e65744ea339ffb8b5
2025-09-07T07:36:04.3682778Z  * [new tag]                 trunk/2fa0520a64ed8aa734a56c4d124958f0b5711ca8 -> trunk/2fa0520a64ed8aa734a56c4d124958f0b5711ca8
2025-09-07T07:36:04.3683699Z  * [new tag]                 trunk/302df2ac5dc4222294c09d48804a2dddb8f4bad8 -> trunk/302df2ac5dc4222294c09d48804a2dddb8f4bad8
2025-09-07T07:36:04.3684516Z  * [new tag]                 trunk/33028597bfa2e0178e28c8cce33cb9b3800cac43 -> trunk/33028597bfa2e0178e28c8cce33cb9b3800cac43
2025-09-07T07:36:04.3685517Z  * [new tag]                 trunk/34aa78274d6770086025a967fa63a86830e08176 -> trunk/34aa78274d6770086025a967fa63a86830e08176
2025-09-07T07:36:04.3686442Z  * [new tag]                 trunk/3559c354ce6a14d11fe29fb12fa2747a2f2af449 -> trunk/3559c354ce6a14d11fe29fb12fa2747a2f2af449
2025-09-07T07:36:04.3687804Z  * [new tag]                 trunk/36d207fcaaede0d1e58a5168084c307b32b6fd8b -> trunk/36d207fcaaede0d1e58a5168084c307b32b6fd8b
2025-09-07T07:36:04.3688605Z  * [new tag]                 trunk/377033757ae5ca524ea842f1b0a5f446ed3d8fe0 -> trunk/377033757ae5ca524ea842f1b0a5f446ed3d8fe0
2025-09-07T07:36:04.3689655Z  * [new tag]                 trunk/3771380f83fcac154a7c89ad679311d8c4818287 -> trunk/3771380f83fcac154a7c89ad679311d8c4818287
2025-09-07T07:36:04.3690583Z  * [new tag]                 trunk/3a207816cc569f78863d86c01f2a3d265350e39f -> trunk/3a207816cc569f78863d86c01f2a3d265350e39f
2025-09-07T07:36:04.3691531Z  * [new tag]                 trunk/3a20a20e7065ec927fdd216d4da3b04f879b3c67 -> trunk/3a20a20e7065ec927fdd216d4da3b04f879b3c67
2025-09-07T07:36:04.3692546Z  * [new tag]                 trunk/3bbc2e3e4f025523eaa5dbff220b3e96bca608d0 -> trunk/3bbc2e3e4f025523eaa5dbff220b3e96bca608d0
2025-09-07T07:36:04.3693503Z  * [new tag]                 trunk/3c0ff1b569c45cfa6935ad8031a9d4cf1551aa3f -> trunk/3c0ff1b569c45cfa6935ad8031a9d4cf1551aa3f
2025-09-07T07:36:04.3694466Z  * [new tag]                 trunk/3c45af079afc92a03b03ddf4f9198902ffcf30cf -> trunk/3c45af079afc92a03b03ddf4f9198902ffcf30cf
2025-09-07T07:36:04.3695461Z  * [new tag]                 trunk/3dde5d7f9bf80dd6623a712bc429e9e4302464b5 -> trunk/3dde5d7f9bf80dd6623a712bc429e9e4302464b5
2025-09-07T07:36:04.3697530Z  * [new tag]                 trunk/403a3a393cda7e60f503f3b04b8805a845dcf45d -> trunk/403a3a393cda7e60f503f3b04b8805a845dcf45d
2025-09-07T07:36:04.3698097Z  * [new tag]                 trunk/420c52ecf36f86d32da0853bfbe074b682b070aa -> trunk/420c52ecf36f86d32da0853bfbe074b682b070aa
2025-09-07T07:36:04.3699945Z  * [new tag]                 trunk/43b7c86a2c0f91320f5c5f4827b111edff06fdb6 -> trunk/43b7c86a2c0f91320f5c5f4827b111edff06fdb6
2025-09-07T07:36:04.3700462Z  * [new tag]                 trunk/451ed931562ec8b46d1f7e6c266a68132a119336 -> trunk/451ed931562ec8b46d1f7e6c266a68132a119336
2025-09-07T07:36:04.3700959Z  * [new tag]                 trunk/480c7391126656154318fabf1d57ebc01e196e63 -> trunk/480c7391126656154318fabf1d57ebc01e196e63
2025-09-07T07:36:04.3701469Z  * [new tag]                 trunk/48bedd753da22634aa94fbafeb731e82025404f3 -> trunk/48bedd753da22634aa94fbafeb731e82025404f3
2025-09-07T07:36:04.3702094Z  * [new tag]                 trunk/494878a11b79071ada0b98f34042d47155be6d1c -> trunk/494878a11b79071ada0b98f34042d47155be6d1c
2025-09-07T07:36:04.3703170Z  * [new tag]                 trunk/4ae57d448c0a7d37e4cfd5c27d977fad2cef4051 -> trunk/4ae57d448c0a7d37e4cfd5c27d977fad2cef4051
2025-09-07T07:36:04.3704225Z  * [new tag]                 trunk/4cdaf8265d86f984254b62052da8c26ef61ef1cf -> trunk/4cdaf8265d86f984254b62052da8c26ef61ef1cf
2025-09-07T07:36:04.3705133Z  * [new tag]                 trunk/4d4abec80f03cd8fdefe1d9cb3a60d3690cd777e -> trunk/4d4abec80f03cd8fdefe1d9cb3a60d3690cd777e
2025-09-07T07:36:04.3706154Z  * [new tag]                 trunk/4e42aa8ffc44b8340eb0eeaf80a2cafc4763a186 -> trunk/4e42aa8ffc44b8340eb0eeaf80a2cafc4763a186
2025-09-07T07:36:04.3707087Z  * [new tag]                 trunk/4f72d932feee0749397fec876dcd43994f50b215 -> trunk/4f72d932feee0749397fec876dcd43994f50b215
2025-09-07T07:36:04.3708074Z  * [new tag]                 trunk/50fc22dedf3c4a27be61fa05551c4f320281b42d -> trunk/50fc22dedf3c4a27be61fa05551c4f320281b42d
2025-09-07T07:36:04.3709040Z  * [new tag]                 trunk/5211f1f908907ffc064b56e43cf8659f7fc22aa9 -> trunk/5211f1f908907ffc064b56e43cf8659f7fc22aa9
2025-09-07T07:36:04.3710034Z  * [new tag]                 trunk/524b78d4f67045b83bb69edc56ab16efe282971c -> trunk/524b78d4f67045b83bb69edc56ab16efe282971c
2025-09-07T07:36:04.3711046Z  * [new tag]                 trunk/54e275e0d81fe1e1ccfa4fb5f2a5a9aaca00ca15 -> trunk/54e275e0d81fe1e1ccfa4fb5f2a5a9aaca00ca15
2025-09-07T07:36:04.3711910Z  * [new tag]                 trunk/5561e45758d59c94605873d5db48ed459c004c3b -> trunk/5561e45758d59c94605873d5db48ed459c004c3b
2025-09-07T07:36:04.3712951Z  * [new tag]                 trunk/57278d45f046d4f89f45d373b1af4dd56934ff24 -> trunk/57278d45f046d4f89f45d373b1af4dd56934ff24
2025-09-07T07:36:04.3714018Z  * [new tag]                 trunk/5927a70934ccf7b70182d364c23245a7dd685503 -> trunk/5927a70934ccf7b70182d364c23245a7dd685503
2025-09-07T07:36:04.3714971Z  * [new tag]                 trunk/5985e28912aeb40b103ebfcf2fd0665eb4a50599 -> trunk/5985e28912aeb40b103ebfcf2fd0665eb4a50599
2025-09-07T07:36:04.3715957Z  * [new tag]                 trunk/5a2da090ed6db88bb657c4e51ec0b310cd08bff6 -> trunk/5a2da090ed6db88bb657c4e51ec0b310cd08bff6
2025-09-07T07:36:04.3716905Z  * [new tag]                 trunk/5c473e9f5ee0ef0fc38e6cf34a95b547f8cdc8d5 -> trunk/5c473e9f5ee0ef0fc38e6cf34a95b547f8cdc8d5
2025-09-07T07:36:04.3718153Z  * [new tag]                 trunk/5c67426d6847667a7c55a2dd01f470fa37238c18 -> trunk/5c67426d6847667a7c55a2dd01f470fa37238c18
2025-09-07T07:36:04.3719053Z  * [new tag]                 trunk/5da573c42c332bc68d4b7946c69f690a876d951a -> trunk/5da573c42c332bc68d4b7946c69f690a876d951a
2025-09-07T07:36:04.3720024Z  * [new tag]                 trunk/5e5870e858f60ff4bf87d03f3592097e934a9580 -> trunk/5e5870e858f60ff4bf87d03f3592097e934a9580
2025-09-07T07:36:04.3720992Z  * [new tag]                 trunk/5f3cbc9442aa55b5afb29f4ac8ca9be569003e84 -> trunk/5f3cbc9442aa55b5afb29f4ac8ca9be569003e84
2025-09-07T07:36:04.3722054Z  * [new tag]                 trunk/600c25e9a17fe56e3dee872be8854db08916ba0c -> trunk/600c25e9a17fe56e3dee872be8854db08916ba0c
2025-09-07T07:36:04.3722899Z  * [new tag]                 trunk/601ae8e4831fc8123fffcfb8fd2e6b6381b42e14 -> trunk/601ae8e4831fc8123fffcfb8fd2e6b6381b42e14
2025-09-07T07:36:04.3723922Z  * [new tag]                 trunk/6087ef41e54c2494b117ffd923faf20f515a6806 -> trunk/6087ef41e54c2494b117ffd923faf20f515a6806
2025-09-07T07:36:04.3724878Z  * [new tag]                 trunk/626cb7df8161dd4ecb4fe43b60f37ce9076f56b1 -> trunk/626cb7df8161dd4ecb4fe43b60f37ce9076f56b1
2025-09-07T07:36:04.3725814Z  * [new tag]                 trunk/62c3f9a97fd3dea7132a93066d32d893ffe101e6 -> trunk/62c3f9a97fd3dea7132a93066d32d893ffe101e6
2025-09-07T07:36:04.3726761Z  * [new tag]                 trunk/63a9c23fe99eacfd09610c36dfe8f01b053c1a35 -> trunk/63a9c23fe99eacfd09610c36dfe8f01b053c1a35
2025-09-07T07:36:04.3727724Z  * [new tag]                 trunk/65985937d97505f648b6ed852c3129f2dd08b251 -> trunk/65985937d97505f648b6ed852c3129f2dd08b251
2025-09-07T07:36:04.3729179Z  * [new tag]                 trunk/66f3b4a682a6153517dd23369fdc3289b6494b07 -> trunk/66f3b4a682a6153517dd23369fdc3289b6494b07
2025-09-07T07:36:04.3729941Z  * [new tag]                 trunk/6737e2c996990024187ba620d2764f3b6f6add2c -> trunk/6737e2c996990024187ba620d2764f3b6f6add2c
2025-09-07T07:36:04.3730933Z  * [new tag]                 trunk/67c31dcd364f10072a55f4a30ffd1151c686283a -> trunk/67c31dcd364f10072a55f4a30ffd1151c686283a
2025-09-07T07:36:04.3731952Z  * [new tag]                 trunk/68738beff73e9c3512e18b4edea811a897ce42db -> trunk/68738beff73e9c3512e18b4edea811a897ce42db
2025-09-07T07:36:04.3732973Z  * [new tag]                 trunk/69a25f68884a168550695fdb1a7c310c54d29536 -> trunk/69a25f68884a168550695fdb1a7c310c54d29536
2025-09-07T07:36:04.3733899Z  * [new tag]                 trunk/6b1900c22f1a07b9519346898d4c71d8a2b0f12f -> trunk/6b1900c22f1a07b9519346898d4c71d8a2b0f12f
2025-09-07T07:36:04.3734891Z  * [new tag]                 trunk/6b8b3ac4403f771bd4a8f9a45d93347304148774 -> trunk/6b8b3ac4403f771bd4a8f9a45d93347304148774
2025-09-07T07:36:04.3735786Z  * [new tag]                 trunk/6f7608d603834d6068b2e7a5d59bec3973b6bb1b -> trunk/6f7608d603834d6068b2e7a5d59bec3973b6bb1b
2025-09-07T07:36:04.3736829Z  * [new tag]                 trunk/70d36e047dfb3488fd6335016711a784d810ebda -> trunk/70d36e047dfb3488fd6335016711a784d810ebda
2025-09-07T07:36:04.3737844Z  * [new tag]                 trunk/71992dd805ff9d6763f77214dfe8b0465e88c87b -> trunk/71992dd805ff9d6763f77214dfe8b0465e88c87b
2025-09-07T07:36:04.3738816Z  * [new tag]                 trunk/734ce8eba9c69381f187359bf0fef1d71d84cd20 -> trunk/734ce8eba9c69381f187359bf0fef1d71d84cd20
2025-09-07T07:36:04.3739809Z  * [new tag]                 trunk/73eb4511fb863a37944342b7e92aae706de603c8 -> trunk/73eb4511fb863a37944342b7e92aae706de603c8
2025-09-07T07:36:04.3740809Z  * [new tag]                 trunk/75bc23cfc345bd4c05e7f97c416c4b3d2d1fa64b -> trunk/75bc23cfc345bd4c05e7f97c416c4b3d2d1fa64b
2025-09-07T07:36:04.3741791Z  * [new tag]                 trunk/771f369448321a387f2018535bc8b8b6e5f12fab -> trunk/771f369448321a387f2018535bc8b8b6e5f12fab
2025-09-07T07:36:04.3742842Z  * [new tag]                 trunk/789d4942127143f2adcb53612c058ce4c9a2cf20 -> trunk/789d4942127143f2adcb53612c058ce4c9a2cf20
2025-09-07T07:36:04.3743672Z  * [new tag]                 trunk/791eff96c85678c950888f9da24650083ee673fe -> trunk/791eff96c85678c950888f9da24650083ee673fe
2025-09-07T07:36:04.3744518Z  * [new tag]                 trunk/793fc12aff1f69fbbf9f4278182fb52bbe350fc9 -> trunk/793fc12aff1f69fbbf9f4278182fb52bbe350fc9
2025-09-07T07:36:04.3745440Z  * [new tag]                 trunk/79fcd5247a9a129eee526a14df30bfc6a22b3f01 -> trunk/79fcd5247a9a129eee526a14df30bfc6a22b3f01
2025-09-07T07:36:04.3746385Z  * [new tag]                 trunk/7f4ff79210eb06924f223ae3a1941ee0e2635348 -> trunk/7f4ff79210eb06924f223ae3a1941ee0e2635348
2025-09-07T07:36:04.3747402Z  * [new tag]                 trunk/8076a185c85112be62be292eb47409c88a585b1c -> trunk/8076a185c85112be62be292eb47409c88a585b1c
2025-09-07T07:36:04.3748300Z  * [new tag]                 trunk/80dd397f1979371a5583fa3d5c7352029522a78d -> trunk/80dd397f1979371a5583fa3d5c7352029522a78d
2025-09-07T07:36:04.3749152Z  * [new tag]                 trunk/8171d6052ec12628eb67e0040839314056014429 -> trunk/8171d6052ec12628eb67e0040839314056014429
2025-09-07T07:36:04.3750595Z  * [new tag]                 trunk/81aeefa657b7ccc26b275c50a9f33b2f056e8071 -> trunk/81aeefa657b7ccc26b275c50a9f33b2f056e8071
2025-09-07T07:36:04.3751518Z  * [new tag]                 trunk/81b7b16618bda250ce55982894a83dc0805eb64c -> trunk/81b7b16618bda250ce55982894a83dc0805eb64c
2025-09-07T07:36:04.3752511Z  * [new tag]                 trunk/827f0d405448de31f79d1089f7d7fceab2f87895 -> trunk/827f0d405448de31f79d1089f7d7fceab2f87895
2025-09-07T07:36:04.3753505Z  * [new tag]                 trunk/82f63c8f6de63c30132a8ac299b6e8c2fd0d3fe8 -> trunk/82f63c8f6de63c30132a8ac299b6e8c2fd0d3fe8
2025-09-07T07:36:04.3754473Z  * [new tag]                 trunk/850e1382a9c56bfde18af09d3e72352d775e9435 -> trunk/850e1382a9c56bfde18af09d3e72352d775e9435
2025-09-07T07:36:04.3755486Z  * [new tag]                 trunk/8678d831c48e616b717bff50f2d03141d2e9f965 -> trunk/8678d831c48e616b717bff50f2d03141d2e9f965
2025-09-07T07:36:04.3756518Z  * [new tag]                 trunk/869cbcc16e489a4f5a14a93d5779b0ea86061c60 -> trunk/869cbcc16e489a4f5a14a93d5779b0ea86061c60
2025-09-07T07:36:04.3757527Z  * [new tag]                 trunk/8703debf669bc2238211bfd039f4ecdd8228b7f7 -> trunk/8703debf669bc2238211bfd039f4ecdd8228b7f7
2025-09-07T07:36:04.3758495Z  * [new tag]                 trunk/874069fbe46e82da5cfa405e6c0deb12e89ff608 -> trunk/874069fbe46e82da5cfa405e6c0deb12e89ff608
2025-09-07T07:36:04.3759544Z  * [new tag]                 trunk/8875d6e394da2fffd04f31b28bf258c94d4776a3 -> trunk/8875d6e394da2fffd04f31b28bf258c94d4776a3
2025-09-07T07:36:04.3760542Z  * [new tag]                 trunk/88d94d17e8c5155451393afa6eb3bab48ab61c16 -> trunk/88d94d17e8c5155451393afa6eb3bab48ab61c16
2025-09-07T07:36:04.3761549Z  * [new tag]                 trunk/890626632def7e0ef95a2d01e87a0e4627824a9f -> trunk/890626632def7e0ef95a2d01e87a0e4627824a9f
2025-09-07T07:36:04.3762608Z  * [new tag]                 trunk/8975cda2520b7b1b5bc3b4d8213edf261fa82570 -> trunk/8975cda2520b7b1b5bc3b4d8213edf261fa82570
2025-09-07T07:36:04.3763626Z  * [new tag]                 trunk/89d41d3f61d04f14730ec26f008a59bef6624610 -> trunk/89d41d3f61d04f14730ec26f008a59bef6624610
2025-09-07T07:36:04.3764626Z  * [new tag]                 trunk/8bb213b6d599ef1273fe52f9b1f6d476056c3a41 -> trunk/8bb213b6d599ef1273fe52f9b1f6d476056c3a41
2025-09-07T07:36:04.3765571Z  * [new tag]                 trunk/8e23a1227b5fb2e39afaa7d57c075a75b640a5af -> trunk/8e23a1227b5fb2e39afaa7d57c075a75b640a5af
2025-09-07T07:36:04.3767062Z  * [new tag]                 trunk/8ec551bb354ab2b85fbbba9d461740a20366d248 -> trunk/8ec551bb354ab2b85fbbba9d461740a20366d248
2025-09-07T07:36:04.3768048Z  * [new tag]                 trunk/8fd3c9ce919c8d5c645fd348bba517e948cbc29d -> trunk/8fd3c9ce919c8d5c645fd348bba517e948cbc29d
2025-09-07T07:36:04.3769360Z  * [new tag]                 trunk/90f50f7e68e120d9574e6e3189e37b4280010ad9 -> trunk/90f50f7e68e120d9574e6e3189e37b4280010ad9
2025-09-07T07:36:04.3770356Z  * [new tag]                 trunk/91f0bcf43fc0bc743350d491ac63b77e92054ac9 -> trunk/91f0bcf43fc0bc743350d491ac63b77e92054ac9
2025-09-07T07:36:04.3771447Z  * [new tag]                 trunk/92576a594b8121f6b0b1b5a3ea16d08792fc68ab -> trunk/92576a594b8121f6b0b1b5a3ea16d08792fc68ab
2025-09-07T07:36:04.3772497Z  * [new tag]                 trunk/92a43025e0baa1f2ce345f28d22913b518a1ab9d -> trunk/92a43025e0baa1f2ce345f28d22913b518a1ab9d
2025-09-07T07:36:04.3773359Z  * [new tag]                 trunk/93fb23d6fae7c4e82c4239a1033e522088742634 -> trunk/93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:36:04.3774364Z  * [new tag]                 trunk/9458d1ac3bd70c2af316a8ba95d2c6c9c1199c9c -> trunk/9458d1ac3bd70c2af316a8ba95d2c6c9c1199c9c
2025-09-07T07:36:04.3775465Z  * [new tag]                 trunk/9480cdc0b61488c89a23c2f64f43b2dcedc8728e -> trunk/9480cdc0b61488c89a23c2f64f43b2dcedc8728e
2025-09-07T07:36:04.3776721Z  * [new tag]                 trunk/9491d289b329e4ba4a9f5f5b1be7960671bb7840 -> trunk/9491d289b329e4ba4a9f5f5b1be7960671bb7840
2025-09-07T07:36:04.3777635Z  * [new tag]                 trunk/9499c8761cd2067feb9877414e818f6fd00290f1 -> trunk/9499c8761cd2067feb9877414e818f6fd00290f1
2025-09-07T07:36:04.3778934Z  * [new tag]                 trunk/95ee0bfea99d3d346d6502b91b497d2b35795504 -> trunk/95ee0bfea99d3d346d6502b91b497d2b35795504
2025-09-07T07:36:04.3779898Z  * [new tag]                 trunk/98374612fc2febd686be20761e56bdc2424bc36a -> trunk/98374612fc2febd686be20761e56bdc2424bc36a
2025-09-07T07:36:04.3780965Z  * [new tag]                 trunk/98efc9e93d8fc61eb53cb91378443617cb550500 -> trunk/98efc9e93d8fc61eb53cb91378443617cb550500
2025-09-07T07:36:04.3781994Z  * [new tag]                 trunk/994f2a5dbcbdc915da39bf6f6ce4d1f5e74835c9 -> trunk/994f2a5dbcbdc915da39bf6f6ce4d1f5e74835c9
2025-09-07T07:36:04.3782998Z  * [new tag]                 trunk/99f356fa58c8d726cef022d8710f5491291158f6 -> trunk/99f356fa58c8d726cef022d8710f5491291158f6
2025-09-07T07:36:04.3783987Z  * [new tag]                 trunk/9a1c5c0a078b94d13ac5c1ae0d754d19fb73bf99 -> trunk/9a1c5c0a078b94d13ac5c1ae0d754d19fb73bf99
2025-09-07T07:36:04.3784972Z  * [new tag]                 trunk/9a665ca3c472384e9d722bddba79e5a7680f1abd -> trunk/9a665ca3c472384e9d722bddba79e5a7680f1abd
2025-09-07T07:36:04.3785984Z  * [new tag]                 trunk/9aedb3cd87b52160872173c177f61053d97bed57 -> trunk/9aedb3cd87b52160872173c177f61053d97bed57
2025-09-07T07:36:04.3786948Z  * [new tag]                 trunk/9b81fe281da41f2421506339d26b027a468902f4 -> trunk/9b81fe281da41f2421506339d26b027a468902f4
2025-09-07T07:36:04.3787966Z  * [new tag]                 trunk/9bdcee01f86e2969cff1140cdecfca13cb51816e -> trunk/9bdcee01f86e2969cff1140cdecfca13cb51816e
2025-09-07T07:36:04.3788994Z  * [new tag]                 trunk/9c03d6be87eedc06e524e202e07a7e776551a839 -> trunk/9c03d6be87eedc06e524e202e07a7e776551a839
2025-09-07T07:36:04.3790035Z  * [new tag]                 trunk/9c957723a0fedd9c637e63e023a613019e2cab60 -> trunk/9c957723a0fedd9c637e63e023a613019e2cab60
2025-09-07T07:36:04.3791009Z  * [new tag]                 trunk/9e5247f51d81735e5f1e65e80588985fa93bccc5 -> trunk/9e5247f51d81735e5f1e65e80588985fa93bccc5
2025-09-07T07:36:04.3792043Z  * [new tag]                 trunk/9eadb37cdd699f7e8e8177a5227bfeb16184ef26 -> trunk/9eadb37cdd699f7e8e8177a5227bfeb16184ef26
2025-09-07T07:36:04.3793037Z  * [new tag]                 trunk/a00cdc1e4159db73c9ffb3f25e93e55877709a29 -> trunk/a00cdc1e4159db73c9ffb3f25e93e55877709a29
2025-09-07T07:36:04.3794027Z  * [new tag]                 trunk/a02ee4a816d11380c6f564c1aba64d56af5ba705 -> trunk/a02ee4a816d11380c6f564c1aba64d56af5ba705
2025-09-07T07:36:04.3795008Z  * [new tag]                 trunk/a3c7f77e50f900721817934120d60c2361b3c40d -> trunk/a3c7f77e50f900721817934120d60c2361b3c40d
2025-09-07T07:36:04.3796051Z  * [new tag]                 trunk/a3d72b09ae12126a2b7d4a63a45ac100a882a802 -> trunk/a3d72b09ae12126a2b7d4a63a45ac100a882a802
2025-09-07T07:36:04.3797034Z  * [new tag]                 trunk/a3e5466002791da609fcb069155d8ee347baee92 -> trunk/a3e5466002791da609fcb069155d8ee347baee92
2025-09-07T07:36:04.3798025Z  * [new tag]                 trunk/a714437093ed196eee28f7de454cf4c41badc098 -> trunk/a714437093ed196eee28f7de454cf4c41badc098
2025-09-07T07:36:04.3798983Z  * [new tag]                 trunk/a75e8cd27098f290de0b7439685d05ce02e91356 -> trunk/a75e8cd27098f290de0b7439685d05ce02e91356
2025-09-07T07:36:04.3800007Z  * [new tag]                 trunk/a8d6943d36c1c2a5f90d3573460695bad4b623ae -> trunk/a8d6943d36c1c2a5f90d3573460695bad4b623ae
2025-09-07T07:36:04.3800818Z  * [new tag]                 trunk/a918bbad6ab20649ff82eefb48417ecbe96bcb34 -> trunk/a918bbad6ab20649ff82eefb48417ecbe96bcb34
2025-09-07T07:36:04.3802015Z  * [new tag]                 trunk/a99d8d39bc842d6ebc3e368b178e4884d24b056e -> trunk/a99d8d39bc842d6ebc3e368b178e4884d24b056e
2025-09-07T07:36:04.3802934Z  * [new tag]                 trunk/aac1a50a191b4102d566c9c1ea22f06d6c2e3f02 -> trunk/aac1a50a191b4102d566c9c1ea22f06d6c2e3f02
2025-09-07T07:36:04.3804160Z  * [new tag]                 trunk/aad96a202244c7d0d120c04ba8db593edd8c0f92 -> trunk/aad96a202244c7d0d120c04ba8db593edd8c0f92
2025-09-07T07:36:04.3805146Z  * [new tag]                 trunk/ab643e4dbbaf7b663d4237514cbf01af9b11565c -> trunk/ab643e4dbbaf7b663d4237514cbf01af9b11565c
2025-09-07T07:36:04.3806173Z  * [new tag]                 trunk/abc447174cd2cf8591edbc70a9f836f9a5779f47 -> trunk/abc447174cd2cf8591edbc70a9f836f9a5779f47
2025-09-07T07:36:04.3807126Z  * [new tag]                 trunk/acece97c3a9dceb63194e314da93fdf37cf15a0d -> trunk/acece97c3a9dceb63194e314da93fdf37cf15a0d
2025-09-07T07:36:04.3808230Z  * [new tag]                 trunk/adae7f66aacf3f248c3101b858cf98d5809119fa -> trunk/adae7f66aacf3f248c3101b858cf98d5809119fa
2025-09-07T07:36:04.3809218Z  * [new tag]                 trunk/ae0edc133e61e3b16caf0b2ee0ff3f33ab72af4c -> trunk/ae0edc133e61e3b16caf0b2ee0ff3f33ab72af4c
2025-09-07T07:36:04.3810171Z  * [new tag]                 trunk/aed33a8fcbd60b052d4559d261390c5797129c6d -> trunk/aed33a8fcbd60b052d4559d261390c5797129c6d
2025-09-07T07:36:04.3811179Z  * [new tag]                 trunk/b04e922712080a3652e438d05e8bb74e0cd2d238 -> trunk/b04e922712080a3652e438d05e8bb74e0cd2d238
2025-09-07T07:36:04.3812222Z  * [new tag]                 trunk/b0a3e58dd71c1a039ac0ef51e5bd8f704f632f6f -> trunk/b0a3e58dd71c1a039ac0ef51e5bd8f704f632f6f
2025-09-07T07:36:04.3813219Z  * [new tag]                 trunk/b16d3f4c8c01d461c2f01064e9ca5fa2b33f5cf1 -> trunk/b16d3f4c8c01d461c2f01064e9ca5fa2b33f5cf1
2025-09-07T07:36:04.3814178Z  * [new tag]                 trunk/b18bb6796f210a183e687d9d64984a5a9d13cf09 -> trunk/b18bb6796f210a183e687d9d64984a5a9d13cf09
2025-09-07T07:36:04.3815124Z  * [new tag]                 trunk/b1bb98ddebdd3e41bf7987372409bdce96ae55de -> trunk/b1bb98ddebdd3e41bf7987372409bdce96ae55de
2025-09-07T07:36:04.3816679Z  * [new tag]                 trunk/b2b4add0e754411372060e1d7b4057a66439172b -> trunk/b2b4add0e754411372060e1d7b4057a66439172b
2025-09-07T07:36:04.3817794Z  * [new tag]                 trunk/b2c7b9ad2dc5a7c0b61febd307761bd5bc2f0f05 -> trunk/b2c7b9ad2dc5a7c0b61febd307761bd5bc2f0f05
2025-09-07T07:36:04.3819000Z  * [new tag]                 trunk/b40d9432be44a6b5974ee62e7d19c3c61c5ece37 -> trunk/b40d9432be44a6b5974ee62e7d19c3c61c5ece37
2025-09-07T07:36:04.3820018Z  * [new tag]                 trunk/b4ad38279b178b7bd14355123c1101e2e853e77b -> trunk/b4ad38279b178b7bd14355123c1101e2e853e77b
2025-09-07T07:36:04.3821048Z  * [new tag]                 trunk/b67c41039835bd9b20b83cd6233e86baaa5f5dde -> trunk/b67c41039835bd9b20b83cd6233e86baaa5f5dde
2025-09-07T07:36:04.3822149Z  * [new tag]                 trunk/b6d0a9ea9056ede4f7024dbf3bd6c43be3aff49c -> trunk/b6d0a9ea9056ede4f7024dbf3bd6c43be3aff49c
2025-09-07T07:36:04.3823147Z  * [new tag]                 trunk/b7dad7dd49448c88d0751fa2e29c70afe985f734 -> trunk/b7dad7dd49448c88d0751fa2e29c70afe985f734
2025-09-07T07:36:04.3824139Z  * [new tag]                 trunk/b7e207ca9f046ddd716076965a0cce403ba99052 -> trunk/b7e207ca9f046ddd716076965a0cce403ba99052
2025-09-07T07:36:04.3825122Z  * [new tag]                 trunk/b919560c4a7010e2d89facee25586269a994746e -> trunk/b919560c4a7010e2d89facee25586269a994746e
2025-09-07T07:36:04.3826151Z  * [new tag]                 trunk/b9ba612f7a968f7b27e121ca8f4d0a4d954f5354 -> trunk/b9ba612f7a968f7b27e121ca8f4d0a4d954f5354
2025-09-07T07:36:04.3827213Z  * [new tag]                 trunk/ba7f546ccccb5e0b36d9070dc25f26a9647f89f8 -> trunk/ba7f546ccccb5e0b36d9070dc25f26a9647f89f8
2025-09-07T07:36:04.3828211Z  * [new tag]                 trunk/bb950284c7e72905994bc25dd436c10e48088d85 -> trunk/bb950284c7e72905994bc25dd436c10e48088d85
2025-09-07T07:36:04.3829200Z  * [new tag]                 trunk/bbedc71fd3267c639c38b4ec25eaa22f973d9c4d -> trunk/bbedc71fd3267c639c38b4ec25eaa22f973d9c4d
2025-09-07T07:36:04.3830111Z  * [new tag]                 trunk/bc4db2c27fce6ff1648bdc5af31ec225d2a31f37 -> trunk/bc4db2c27fce6ff1648bdc5af31ec225d2a31f37
2025-09-07T07:36:04.3831043Z  * [new tag]                 trunk/bc505977fb66677a09c31155c987330fbb18a865 -> trunk/bc505977fb66677a09c31155c987330fbb18a865
2025-09-07T07:36:04.3832027Z  * [new tag]                 trunk/bd39e47feea7326afb5bbb67fcb1e69279239527 -> trunk/bd39e47feea7326afb5bbb67fcb1e69279239527
2025-09-07T07:36:04.3833125Z  * [new tag]                 trunk/be5b03dde96638f25ffd732a4fed7e41b4cf40e1 -> trunk/be5b03dde96638f25ffd732a4fed7e41b4cf40e1
2025-09-07T07:36:04.3834117Z  * [new tag]                 trunk/bffc7dd1f374d8408911cd22c6b3d6df39ded9b3 -> trunk/bffc7dd1f374d8408911cd22c6b3d6df39ded9b3
2025-09-07T07:36:04.3835208Z  * [new tag]                 trunk/c024b1f5a18d5c5aee5cc2acdd4c52b24b93ffcf -> trunk/c024b1f5a18d5c5aee5cc2acdd4c52b24b93ffcf
2025-09-07T07:36:04.3836190Z  * [new tag]                 trunk/c0983e6cc0acf71689e1851d12609e00b3f59371 -> trunk/c0983e6cc0acf71689e1851d12609e00b3f59371
2025-09-07T07:36:04.3837169Z  * [new tag]                 trunk/c10195e723eeeedd099ed8b73eda7184ca618fad -> trunk/c10195e723eeeedd099ed8b73eda7184ca618fad
2025-09-07T07:36:04.3838186Z  * [new tag]                 trunk/c157cf6488ade6a7ee2ce2d25b059e1335630a99 -> trunk/c157cf6488ade6a7ee2ce2d25b059e1335630a99
2025-09-07T07:36:04.3839190Z  * [new tag]                 trunk/c2a30246172fd71d56529907ffd3c27b76b1f3a7 -> trunk/c2a30246172fd71d56529907ffd3c27b76b1f3a7
2025-09-07T07:36:04.3840181Z  * [new tag]                 trunk/c32111149921b48bfef909293f1049e21619ed76 -> trunk/c32111149921b48bfef909293f1049e21619ed76
2025-09-07T07:36:04.3840997Z  * [new tag]                 trunk/c37103234afc832dcad307e9016230810957c9d5 -> trunk/c37103234afc832dcad307e9016230810957c9d5
2025-09-07T07:36:04.3842035Z  * [new tag]                 trunk/c3ceca2995cd35e1376c4b0704669bff1a81e836 -> trunk/c3ceca2995cd35e1376c4b0704669bff1a81e836
2025-09-07T07:36:04.3843041Z  * [new tag]                 trunk/c3d54dea9febb1236d48d19e5d4876a63f2e20fd -> trunk/c3d54dea9febb1236d48d19e5d4876a63f2e20fd
2025-09-07T07:36:04.3844132Z  * [new tag]                 trunk/c465b3d52c5687fe910d35a5c75341b77f821741 -> trunk/c465b3d52c5687fe910d35a5c75341b77f821741
2025-09-07T07:36:04.3845150Z  * [new tag]                 trunk/c5b8a10be5e89396da916d1069ffcb7135f0372b -> trunk/c5b8a10be5e89396da916d1069ffcb7135f0372b
2025-09-07T07:36:04.3846105Z  * [new tag]                 trunk/c7e41071a08f4045bc11ab60ec366d7357d56e30 -> trunk/c7e41071a08f4045bc11ab60ec366d7357d56e30
2025-09-07T07:36:04.3847051Z  * [new tag]                 trunk/c98ddaca6d2e19ca37aff00c4ff0cda1e9a6ff65 -> trunk/c98ddaca6d2e19ca37aff00c4ff0cda1e9a6ff65
2025-09-07T07:36:04.3848010Z  * [new tag]                 trunk/cb1e31362c7b53acf4ac95b9f8878064c184f03b -> trunk/cb1e31362c7b53acf4ac95b9f8878064c184f03b
2025-09-07T07:36:04.3848993Z  * [new tag]                 trunk/cbfb005f7cce79974795b148e265f594f59477c8 -> trunk/cbfb005f7cce79974795b148e265f594f59477c8
2025-09-07T07:36:04.3850057Z  * [new tag]                 trunk/cc5bdd12401bda835291d2f3cb297132ebdbf358 -> trunk/cc5bdd12401bda835291d2f3cb297132ebdbf358
2025-09-07T07:36:04.3851163Z  * [new tag]                 trunk/cd529b686d54bbaa443f5b310140de48422d96c7 -> trunk/cd529b686d54bbaa443f5b310140de48422d96c7
2025-09-07T07:36:04.3852119Z  * [new tag]                 trunk/cec0ff122815582af5302360aff03676558c5c87 -> trunk/cec0ff122815582af5302360aff03676558c5c87
2025-09-07T07:36:04.3853126Z  * [new tag]                 trunk/d11720efdb563d02cf4f7d324311fb15a755268e -> trunk/d11720efdb563d02cf4f7d324311fb15a755268e
2025-09-07T07:36:04.3854107Z  * [new tag]                 trunk/d1706d9128ae24d9048167e80d3fe5196d19035e -> trunk/d1706d9128ae24d9048167e80d3fe5196d19035e
2025-09-07T07:36:04.3855113Z  * [new tag]                 trunk/d1a15abfdcaef138f2d9e93a9f46be44f30b766d -> trunk/d1a15abfdcaef138f2d9e93a9f46be44f30b766d
2025-09-07T07:36:04.3856316Z  * [new tag]                 trunk/d232a95d4a79404ca05c1f52d37fde7339dcdf49 -> trunk/d232a95d4a79404ca05c1f52d37fde7339dcdf49
2025-09-07T07:36:04.3857289Z  * [new tag]                 trunk/d2d4c8e9b2371c9aacfb771d9402ac7427b9778e -> trunk/d2d4c8e9b2371c9aacfb771d9402ac7427b9778e
2025-09-07T07:36:04.3858353Z  * [new tag]                 trunk/d33840c542b387ab08ba49aa6c45aa9567fd9be7 -> trunk/d33840c542b387ab08ba49aa6c45aa9567fd9be7
2025-09-07T07:36:04.3859330Z  * [new tag]                 trunk/d5643e8f3a648a99636bfa1f2a41d54bd3c0d0f1 -> trunk/d5643e8f3a648a99636bfa1f2a41d54bd3c0d0f1
2025-09-07T07:36:04.3860295Z  * [new tag]                 trunk/d5b38410b5b6cf75c7a7389972777a6497926ee7 -> trunk/d5b38410b5b6cf75c7a7389972777a6497926ee7
2025-09-07T07:36:04.3861231Z  * [new tag]                 trunk/d5e0f4202ba14632e4d14862ace096609e763462 -> trunk/d5e0f4202ba14632e4d14862ace096609e763462
2025-09-07T07:36:04.3862220Z  * [new tag]                 trunk/d636c181f9140a7b59be10b36eae23039fc2bb72 -> trunk/d636c181f9140a7b59be10b36eae23039fc2bb72
2025-09-07T07:36:04.3863607Z  * [new tag]                 trunk/d64718503728001a1e78168fd7f2d4ff23e57285 -> trunk/d64718503728001a1e78168fd7f2d4ff23e57285
2025-09-07T07:36:04.3864660Z  * [new tag]                 trunk/d67c29ad22670320d676b02e394274af34e8e643 -> trunk/d67c29ad22670320d676b02e394274af34e8e643
2025-09-07T07:36:04.3865653Z  * [new tag]                 trunk/d6b74568e2c98ce58ecc145b72ac66d4caf7ce95 -> trunk/d6b74568e2c98ce58ecc145b72ac66d4caf7ce95
2025-09-07T07:36:04.3866722Z  * [new tag]                 trunk/d711f27845abd45007ccab6076649ebd896c2661 -> trunk/d711f27845abd45007ccab6076649ebd896c2661
2025-09-07T07:36:04.3867797Z  * [new tag]                 trunk/d9d6dde0f42d4bcc8c97671ac50d5096c7e500ab -> trunk/d9d6dde0f42d4bcc8c97671ac50d5096c7e500ab
2025-09-07T07:36:04.3869001Z  * [new tag]                 trunk/da4db4b33d1fdd046650cf19fdbac581a19bf2f9 -> trunk/da4db4b33d1fdd046650cf19fdbac581a19bf2f9
2025-09-07T07:36:04.3869846Z  * [new tag]                 trunk/dac8a4b91c01c3bbc96f54e621b1ea4ffdbd29d1 -> trunk/dac8a4b91c01c3bbc96f54e621b1ea4ffdbd29d1
2025-09-07T07:36:04.3870974Z  * [new tag]                 trunk/dbec08729fb9848bebed6048c63831b87170d061 -> trunk/dbec08729fb9848bebed6048c63831b87170d061
2025-09-07T07:36:04.3871806Z  * [new tag]                 trunk/dcf385395d838f38c8dca25913578230dd43099a -> trunk/dcf385395d838f38c8dca25913578230dd43099a
2025-09-07T07:36:04.3872835Z  * [new tag]                 trunk/dd2519abe83ec3c40d4797492434e41fe3b47e17 -> trunk/dd2519abe83ec3c40d4797492434e41fe3b47e17
2025-09-07T07:36:04.3873829Z  * [new tag]                 trunk/dec72ea4b006dd0fbcaaaa106ad273d73807ab9d -> trunk/dec72ea4b006dd0fbcaaaa106ad273d73807ab9d
2025-09-07T07:36:04.3874800Z  * [new tag]                 trunk/e0a62b266c021b910ce6dc02a6c9429210487717 -> trunk/e0a62b266c021b910ce6dc02a6c9429210487717
2025-09-07T07:36:04.3875835Z  * [new tag]                 trunk/e19e02c84c9dcc408375e5cae3b0709c18b99228 -> trunk/e19e02c84c9dcc408375e5cae3b0709c18b99228
2025-09-07T07:36:04.3876923Z  * [new tag]                 trunk/e304ea4e69d3a7deeb7e48c7450c214a4c953937 -> trunk/e304ea4e69d3a7deeb7e48c7450c214a4c953937
2025-09-07T07:36:04.3877943Z  * [new tag]                 trunk/e3068cdb446adefb5a875616ba37a60235391439 -> trunk/e3068cdb446adefb5a875616ba37a60235391439
2025-09-07T07:36:04.3878933Z  * [new tag]                 trunk/e381d4b0205d5f126c1de534f867ba776f7c3ee6 -> trunk/e381d4b0205d5f126c1de534f867ba776f7c3ee6
2025-09-07T07:36:04.3879988Z  * [new tag]                 trunk/e4bd0ff4f8981b805df32ea5b3550621965ea4f2 -> trunk/e4bd0ff4f8981b805df32ea5b3550621965ea4f2
2025-09-07T07:36:04.3880972Z  * [new tag]                 trunk/e532c9d4f1cdcbc1ea9628f55b9813e77847bdc7 -> trunk/e532c9d4f1cdcbc1ea9628f55b9813e77847bdc7
2025-09-07T07:36:04.3882551Z  * [new tag]                 trunk/e92cd9415377403b6e90585e764639e2e0b5973b -> trunk/e92cd9415377403b6e90585e764639e2e0b5973b
2025-09-07T07:36:04.3883751Z  * [new tag]                 trunk/e9481b6617b5576b099d8ca5798111592e9ad090 -> trunk/e9481b6617b5576b099d8ca5798111592e9ad090
2025-09-07T07:36:04.3884499Z  * [new tag]                 trunk/ea1883dfd3e42defe37b11202b878bb76defa087 -> trunk/ea1883dfd3e42defe37b11202b878bb76defa087
2025-09-07T07:36:04.3885628Z  * [new tag]                 trunk/eac3d6f04cfbbebe3d470dacd216da7d4b1f95a8 -> trunk/eac3d6f04cfbbebe3d470dacd216da7d4b1f95a8
2025-09-07T07:36:04.3886568Z  * [new tag]                 trunk/eb18d32bda75189494d955aa001ade15f10333de -> trunk/eb18d32bda75189494d955aa001ade15f10333de
2025-09-07T07:36:04.3887386Z  * [new tag]                 trunk/ef3be6726f7ff4b77c22db10cec5b686f9107ea9 -> trunk/ef3be6726f7ff4b77c22db10cec5b686f9107ea9
2025-09-07T07:36:04.3888419Z  * [new tag]                 trunk/ef8aabd42422725026cb4dbf48aafa9efa226a04 -> trunk/ef8aabd42422725026cb4dbf48aafa9efa226a04
2025-09-07T07:36:04.3889506Z  * [new tag]                 trunk/f00445b43eee57e20bb9316fa796ca23bf73373b -> trunk/f00445b43eee57e20bb9316fa796ca23bf73373b
2025-09-07T07:36:04.3890534Z  * [new tag]                 trunk/f0c391102b754e3b145e8c59231d2df563487e37 -> trunk/f0c391102b754e3b145e8c59231d2df563487e37
2025-09-07T07:36:04.3891624Z  * [new tag]                 trunk/f27985b7e796fb66a1b476284ba42d8cb360a751 -> trunk/f27985b7e796fb66a1b476284ba42d8cb360a751
2025-09-07T07:36:04.3892692Z  * [new tag]                 trunk/f36f285953700f971552083a5da9d0ceacb63bbd -> trunk/f36f285953700f971552083a5da9d0ceacb63bbd
2025-09-07T07:36:04.3893678Z  * [new tag]                 trunk/f3cebec39ebc110e1c8b06e741896585f7892dbb -> trunk/f3cebec39ebc110e1c8b06e741896585f7892dbb
2025-09-07T07:36:04.3894505Z  * [new tag]                 trunk/f4c33cd44acac92c0b451a04da20ebe9370e5b0c -> trunk/f4c33cd44acac92c0b451a04da20ebe9370e5b0c
2025-09-07T07:36:04.3895612Z  * [new tag]                 trunk/f612045ce105f008b2b675e2fc870163babeb2e8 -> trunk/f612045ce105f008b2b675e2fc870163babeb2e8
2025-09-07T07:36:04.3896619Z  * [new tag]                 trunk/f8746b878dfc1e9639d42cbde832e9b9e792c86c -> trunk/f8746b878dfc1e9639d42cbde832e9b9e792c86c
2025-09-07T07:36:04.3897679Z  * [new tag]                 trunk/f8ffa9194e26523e5f976d4a824d5cc58922727c -> trunk/f8ffa9194e26523e5f976d4a824d5cc58922727c
2025-09-07T07:36:04.3898664Z  * [new tag]                 trunk/f981a7fa5230b98974291fdde32fe8488bc5d469 -> trunk/f981a7fa5230b98974291fdde32fe8488bc5d469
2025-09-07T07:36:04.3899696Z  * [new tag]                 trunk/fbf3d2027daabbcb44d0af274b139be2a248a4f7 -> trunk/fbf3d2027daabbcb44d0af274b139be2a248a4f7
2025-09-07T07:36:04.3900994Z  * [new tag]                 trunk/fca2601c9d628e1bd2d75c7318cd22c4e8c832aa -> trunk/fca2601c9d628e1bd2d75c7318cd22c4e8c832aa
2025-09-07T07:36:04.3902028Z  * [new tag]                 trunk/fea20775ad96bdca972a1811d7d3372f368614ab -> trunk/fea20775ad96bdca972a1811d7d3372f368614ab
2025-09-07T07:36:04.3902861Z  * [new tag]                 trunk/fefee081642f87419a21dc852f7167d4640443cd -> trunk/fefee081642f87419a21dc852f7167d4640443cd
2025-09-07T07:36:04.3904018Z  * [new tag]                 v0.1.1                      -> v0.1.1
2025-09-07T07:36:04.3905052Z  * [new tag]                 v0.1.10                     -> v0.1.10
2025-09-07T07:36:04.3905909Z  * [new tag]                 v0.1.11                     -> v0.1.11
2025-09-07T07:36:04.3906820Z  * [new tag]                 v0.1.12                     -> v0.1.12
2025-09-07T07:36:04.3907694Z  * [new tag]                 v0.1.2                      -> v0.1.2
2025-09-07T07:36:04.3908567Z  * [new tag]                 v0.1.3                      -> v0.1.3
2025-09-07T07:36:04.3909476Z  * [new tag]                 v0.1.4                      -> v0.1.4
2025-09-07T07:36:04.3910383Z  * [new tag]                 v0.1.5                      -> v0.1.5
2025-09-07T07:36:04.3911283Z  * [new tag]                 v0.1.6                      -> v0.1.6
2025-09-07T07:36:04.3912144Z  * [new tag]                 v0.1.7                      -> v0.1.7
2025-09-07T07:36:04.3912993Z  * [new tag]                 v0.1.8                      -> v0.1.8
2025-09-07T07:36:04.3913968Z  * [new tag]                 v0.1.9                      -> v0.1.9
2025-09-07T07:36:04.3914833Z  * [new tag]                 v0.2.0                      -> v0.2.0
2025-09-07T07:36:04.3915767Z  * [new tag]                 v0.3.0                      -> v0.3.0
2025-09-07T07:36:04.3916781Z  * [new tag]                 v0.3.1                      -> v0.3.1
2025-09-07T07:36:04.3917794Z  * [new tag]                 v0.4.0                      -> v0.4.0
2025-09-07T07:36:04.3918911Z  * [new tag]                 v0.4.1                      -> v0.4.1
2025-09-07T07:36:04.3919858Z  * [new tag]                 v1.0.0                      -> v1.0.0
2025-09-07T07:36:04.3920833Z  * [new tag]                 v1.0.0a0                    -> v1.0.0a0
2025-09-07T07:36:04.3921746Z  * [new tag]                 v1.0.1                      -> v1.0.1
2025-09-07T07:36:04.3922711Z  * [new tag]                 v1.0rc0                     -> v1.0rc0
2025-09-07T07:36:04.3923471Z  * [new tag]                 v1.0rc1                     -> v1.0rc1
2025-09-07T07:36:04.3924408Z  * [new tag]                 v1.1.0                      -> v1.1.0
2025-09-07T07:36:04.3925363Z  * [new tag]                 v1.1.0a0                    -> v1.1.0a0
2025-09-07T07:36:04.3926474Z  * [new tag]                 v1.10.0                     -> v1.10.0
2025-09-07T07:36:04.3927500Z  * [new tag]                 v1.10.0-rc1                 -> v1.10.0-rc1
2025-09-07T07:36:04.3928454Z  * [new tag]                 v1.10.0-rc2                 -> v1.10.0-rc2
2025-09-07T07:36:04.3929653Z  * [new tag]                 v1.10.0-rc3                 -> v1.10.0-rc3
2025-09-07T07:36:04.3930630Z  * [new tag]                 v1.10.1                     -> v1.10.1
2025-09-07T07:36:04.3931448Z  * [new tag]                 v1.10.1-rc1                 -> v1.10.1-rc1
2025-09-07T07:36:04.3932218Z  * [new tag]                 v1.10.2                     -> v1.10.2
2025-09-07T07:36:04.3932905Z  * [new tag]                 v1.10.2-rc1                 -> v1.10.2-rc1
2025-09-07T07:36:04.3933925Z  * [new tag]                 v1.11.0                     -> v1.11.0
2025-09-07T07:36:04.3934959Z  * [new tag]                 v1.11.0-rc1                 -> v1.11.0-rc1
2025-09-07T07:36:04.3936001Z  * [new tag]                 v1.11.0-rc2                 -> v1.11.0-rc2
2025-09-07T07:36:04.3936954Z  * [new tag]                 v1.11.0-rc3                 -> v1.11.0-rc3
2025-09-07T07:36:04.3937975Z  * [new tag]                 v1.11.0-rc4                 -> v1.11.0-rc4
2025-09-07T07:36:04.3938983Z  * [new tag]                 v1.11.0-rc5                 -> v1.11.0-rc5
2025-09-07T07:36:04.3939808Z  * [new tag]                 v1.11.0-rc6                 -> v1.11.0-rc6
2025-09-07T07:36:04.3940454Z  * [new tag]                 v1.11.0-rc7                 -> v1.11.0-rc7
2025-09-07T07:36:04.3941475Z  * [new tag]                 v1.12.0                     -> v1.12.0
2025-09-07T07:36:04.3942458Z  * [new tag]                 v1.12.0-rc1                 -> v1.12.0-rc1
2025-09-07T07:36:04.3943404Z  * [new tag]                 v1.12.0-rc2                 -> v1.12.0-rc2
2025-09-07T07:36:04.3944338Z  * [new tag]                 v1.12.0-rc3                 -> v1.12.0-rc3
2025-09-07T07:36:04.3945303Z  * [new tag]                 v1.12.0-rc4                 -> v1.12.0-rc4
2025-09-07T07:36:04.3946392Z  * [new tag]                 v1.12.0-rc5                 -> v1.12.0-rc5
2025-09-07T07:36:04.3947365Z  * [new tag]                 v1.12.0-rc6                 -> v1.12.0-rc6
2025-09-07T07:36:04.3948131Z  * [new tag]                 v1.12.0-rc7                 -> v1.12.0-rc7
2025-09-07T07:36:04.3948806Z  * [new tag]                 v1.12.0-rc8                 -> v1.12.0-rc8
2025-09-07T07:36:04.3949618Z  * [new tag]                 v1.12.1                     -> v1.12.1
2025-09-07T07:36:04.3950635Z  * [new tag]                 v1.12.1-rc1                 -> v1.12.1-rc1
2025-09-07T07:36:04.3951640Z  * [new tag]                 v1.12.1-rc2                 -> v1.12.1-rc2
2025-09-07T07:36:04.3952700Z  * [new tag]                 v1.12.1-rc3                 -> v1.12.1-rc3
2025-09-07T07:36:04.3953752Z  * [new tag]                 v1.12.1-rc4                 -> v1.12.1-rc4
2025-09-07T07:36:04.3954383Z  * [new tag]                 v1.12.1-rc5                 -> v1.12.1-rc5
2025-09-07T07:36:04.3955441Z  * [new tag]                 v1.13.0                     -> v1.13.0
2025-09-07T07:36:04.3956353Z  * [new tag]                 v1.13.0-rc1                 -> v1.13.0-rc1
2025-09-07T07:36:04.3957232Z  * [new tag]                 v1.13.0-rc2                 -> v1.13.0-rc2
2025-09-07T07:36:04.3958165Z  * [new tag]                 v1.13.0-rc3                 -> v1.13.0-rc3
2025-09-07T07:36:04.3959195Z  * [new tag]                 v1.13.0-rc4                 -> v1.13.0-rc4
2025-09-07T07:36:04.3959946Z  * [new tag]                 v1.13.0-rc5                 -> v1.13.0-rc5
2025-09-07T07:36:04.3960806Z  * [new tag]                 v1.13.0-rc6                 -> v1.13.0-rc6
2025-09-07T07:36:04.3961781Z  * [new tag]                 v1.13.1                     -> v1.13.1
2025-09-07T07:36:04.3962561Z  * [new tag]                 v1.13.1-rc1                 -> v1.13.1-rc1
2025-09-07T07:36:04.3963462Z  * [new tag]                 v1.2.0                      -> v1.2.0
2025-09-07T07:36:04.3964506Z  * [new tag]                 v1.2.0a0                    -> v1.2.0a0
2025-09-07T07:36:04.3965585Z  * [new tag]                 v1.3.0                      -> v1.3.0
2025-09-07T07:36:04.3966578Z  * [new tag]                 v1.3.0a0                    -> v1.3.0a0
2025-09-07T07:36:04.3967354Z  * [new tag]                 v1.3.1                      -> v1.3.1
2025-09-07T07:36:04.3968297Z  * [new tag]                 v1.4.0                      -> v1.4.0
2025-09-07T07:36:04.3969215Z  * [new tag]                 v1.4.0a0                    -> v1.4.0a0
2025-09-07T07:36:04.3969973Z  * [new tag]                 v1.4.1                      -> v1.4.1
2025-09-07T07:36:04.3970956Z  * [new tag]                 v1.5.0                      -> v1.5.0
2025-09-07T07:36:04.3971904Z  * [new tag]                 v1.5.0-rc1                  -> v1.5.0-rc1
2025-09-07T07:36:04.3972918Z  * [new tag]                 v1.5.0-rc2                  -> v1.5.0-rc2
2025-09-07T07:36:04.3973913Z  * [new tag]                 v1.5.0-rc3                  -> v1.5.0-rc3
2025-09-07T07:36:04.3974780Z  * [new tag]                 v1.5.0-rc4                  -> v1.5.0-rc4
2025-09-07T07:36:04.3975542Z  * [new tag]                 v1.5.0-rc5                  -> v1.5.0-rc5
2025-09-07T07:36:04.3976513Z  * [new tag]                 v1.5.1                      -> v1.5.1
2025-09-07T07:36:04.3977287Z  * [new tag]                 v1.5.1-rc1                  -> v1.5.1-rc1
2025-09-07T07:36:04.3978115Z  * [new tag]                 v1.6.0                      -> v1.6.0
2025-09-07T07:36:04.3979159Z  * [new tag]                 v1.6.0-rc1                  -> v1.6.0-rc1
2025-09-07T07:36:04.3980109Z  * [new tag]                 v1.6.0-rc2                  -> v1.6.0-rc2
2025-09-07T07:36:04.3981064Z  * [new tag]                 v1.6.0-rc3                  -> v1.6.0-rc3
2025-09-07T07:36:04.3982084Z  * [new tag]                 v1.6.0-rc4                  -> v1.6.0-rc4
2025-09-07T07:36:04.3982962Z  * [new tag]                 v1.6.0-rc5                  -> v1.6.0-rc5
2025-09-07T07:36:04.3983904Z  * [new tag]                 v1.6.0-rc6                  -> v1.6.0-rc6
2025-09-07T07:36:04.3984701Z  * [new tag]                 v1.6.0-rc7                  -> v1.6.0-rc7
2025-09-07T07:36:04.3985802Z  * [new tag]                 v1.7.0                      -> v1.7.0
2025-09-07T07:36:04.3986808Z  * [new tag]                 v1.7.0-rc1                  -> v1.7.0-rc1
2025-09-07T07:36:04.3987936Z  * [new tag]                 v1.7.0-rc2                  -> v1.7.0-rc2
2025-09-07T07:36:04.3989411Z  * [new tag]                 v1.7.0-rc3                  -> v1.7.0-rc3
2025-09-07T07:36:04.3990205Z  * [new tag]                 v1.7.0-rc4                  -> v1.7.0-rc4
2025-09-07T07:36:04.3991232Z  * [new tag]                 v1.7.1                      -> v1.7.1
2025-09-07T07:36:04.3992421Z  * [new tag]                 v1.7.1-rc1                  -> v1.7.1-rc1
2025-09-07T07:36:04.3993309Z  * [new tag]                 v1.7.1-rc2                  -> v1.7.1-rc2
2025-09-07T07:36:04.3994063Z  * [new tag]                 v1.7.1-rc3                  -> v1.7.1-rc3
2025-09-07T07:36:04.3995064Z  * [new tag]                 v1.8.0                      -> v1.8.0
2025-09-07T07:36:04.3995858Z  * [new tag]                 v1.8.0-rc1                  -> v1.8.0-rc1
2025-09-07T07:36:04.3996861Z  * [new tag]                 v1.8.0-rc2                  -> v1.8.0-rc2
2025-09-07T07:36:04.3997786Z  * [new tag]                 v1.8.0-rc3                  -> v1.8.0-rc3
2025-09-07T07:36:04.3998665Z  * [new tag]                 v1.8.0-rc4                  -> v1.8.0-rc4
2025-09-07T07:36:04.3999467Z  * [new tag]                 v1.8.0-rc5                  -> v1.8.0-rc5
2025-09-07T07:36:04.4000127Z  * [new tag]                 v1.8.1                      -> v1.8.1
2025-09-07T07:36:04.4001148Z  * [new tag]                 v1.8.1-rc1                  -> v1.8.1-rc1
2025-09-07T07:36:04.4001943Z  * [new tag]                 v1.8.1-rc2                  -> v1.8.1-rc2
2025-09-07T07:36:04.4002610Z  * [new tag]                 v1.8.1-rc3                  -> v1.8.1-rc3
2025-09-07T07:36:04.4007542Z  * [new tag]                 v1.8.2                      -> v1.8.2
2025-09-07T07:36:04.4008327Z  * [new tag]                 v1.8.2-rc1                  -> v1.8.2-rc1
2025-09-07T07:36:04.4009364Z  * [new tag]                 v1.9.0                      -> v1.9.0
2025-09-07T07:36:04.4010358Z  * [new tag]                 v1.9.0-rc1                  -> v1.9.0-rc1
2025-09-07T07:36:04.4011378Z  * [new tag]                 v1.9.0-rc2                  -> v1.9.0-rc2
2025-09-07T07:36:04.4012353Z  * [new tag]                 v1.9.0-rc3                  -> v1.9.0-rc3
2025-09-07T07:36:04.4013084Z  * [new tag]                 v1.9.0-rc4                  -> v1.9.0-rc4
2025-09-07T07:36:04.4014113Z  * [new tag]                 v1.9.1                      -> v1.9.1
2025-09-07T07:36:04.4015464Z  * [new tag]                 v1.9.1-rc1                  -> v1.9.1-rc1
2025-09-07T07:36:04.4016233Z  * [new tag]                 v1.9.1-rc2                  -> v1.9.1-rc2
2025-09-07T07:36:04.4017213Z  * [new tag]                 v2.0.0                      -> v2.0.0
2025-09-07T07:36:04.4018323Z  * [new tag]                 v2.0.0-rc1                  -> v2.0.0-rc1
2025-09-07T07:36:04.4019342Z  * [new tag]                 v2.0.0-rc2                  -> v2.0.0-rc2
2025-09-07T07:36:04.4020323Z  * [new tag]                 v2.0.0-rc3                  -> v2.0.0-rc3
2025-09-07T07:36:04.4021234Z  * [new tag]                 v2.0.0-rc4                  -> v2.0.0-rc4
2025-09-07T07:36:04.4022264Z  * [new tag]                 v2.0.0-rc5                  -> v2.0.0-rc5
2025-09-07T07:36:04.4023023Z  * [new tag]                 v2.0.0-rc6                  -> v2.0.0-rc6
2025-09-07T07:36:04.4024055Z  * [new tag]                 v2.0.1                      -> v2.0.1
2025-09-07T07:36:04.4025119Z  * [new tag]                 v2.0.1-rc1                  -> v2.0.1-rc1
2025-09-07T07:36:04.4025903Z  * [new tag]                 v2.0.1-rc2                  -> v2.0.1-rc2
2025-09-07T07:36:04.4026798Z  * [new tag]                 v2.0.1-rc3                  -> v2.0.1-rc3
2025-09-07T07:36:04.4027554Z  * [new tag]                 v2.0.1-rc4                  -> v2.0.1-rc4
2025-09-07T07:36:04.4028904Z  * [new tag]                 v2.1.0                      -> v2.1.0
2025-09-07T07:36:04.4029827Z  * [new tag]                 v2.1.0-rc1                  -> v2.1.0-rc1
2025-09-07T07:36:04.4030773Z  * [new tag]                 v2.1.0-rc2                  -> v2.1.0-rc2
2025-09-07T07:36:04.4031810Z  * [new tag]                 v2.1.0-rc3                  -> v2.1.0-rc3
2025-09-07T07:36:04.4032807Z  * [new tag]                 v2.1.0-rc4                  -> v2.1.0-rc4
2025-09-07T07:36:04.4033762Z  * [new tag]                 v2.1.0-rc5                  -> v2.1.0-rc5
2025-09-07T07:36:04.4034634Z  * [new tag]                 v2.1.0-rc6                  -> v2.1.0-rc6
2025-09-07T07:36:04.4035526Z  * [new tag]                 v2.1.1                      -> v2.1.1
2025-09-07T07:36:04.4036524Z  * [new tag]                 v2.1.1-rc1                  -> v2.1.1-rc1
2025-09-07T07:36:04.4037392Z  * [new tag]                 v2.1.1-rc2                  -> v2.1.1-rc2
2025-09-07T07:36:04.4038464Z  * [new tag]                 v2.1.1-rc3                  -> v2.1.1-rc3
2025-09-07T07:36:04.4039427Z  * [new tag]                 v2.1.1-rc4                  -> v2.1.1-rc4
2025-09-07T07:36:04.4040330Z  * [new tag]                 v2.1.1-rc5                  -> v2.1.1-rc5
2025-09-07T07:36:04.4041091Z  * [new tag]                 v2.1.1-rc6                  -> v2.1.1-rc6
2025-09-07T07:36:04.4041999Z  * [new tag]                 v2.1.2                      -> v2.1.2
2025-09-07T07:36:04.4043008Z  * [new tag]                 v2.1.2-rc1                  -> v2.1.2-rc1
2025-09-07T07:36:04.4044030Z  * [new tag]                 v2.1.2-rc2                  -> v2.1.2-rc2
2025-09-07T07:36:04.4044798Z  * [new tag]                 v2.1.2-rc3                  -> v2.1.2-rc3
2025-09-07T07:36:04.4045772Z  * [new tag]                 v2.2.0                      -> v2.2.0
2025-09-07T07:36:04.4046744Z  * [new tag]                 v2.2.0-rc1                  -> v2.2.0-rc1
2025-09-07T07:36:04.4047634Z  * [new tag]                 v2.2.0-rc2                  -> v2.2.0-rc2
2025-09-07T07:36:04.4048548Z  * [new tag]                 v2.2.0-rc3                  -> v2.2.0-rc3
2025-09-07T07:36:04.4049430Z  * [new tag]                 v2.2.0-rc4                  -> v2.2.0-rc4
2025-09-07T07:36:04.4050363Z  * [new tag]                 v2.2.0-rc5                  -> v2.2.0-rc5
2025-09-07T07:36:04.4051258Z  * [new tag]                 v2.2.0-rc6                  -> v2.2.0-rc6
2025-09-07T07:36:04.4052031Z  * [new tag]                 v2.2.0-rc7                  -> v2.2.0-rc7
2025-09-07T07:36:04.4053239Z  * [new tag]                 v2.2.0-rc8                  -> v2.2.0-rc8
2025-09-07T07:36:04.4054282Z  * [new tag]                 v2.2.1                      -> v2.2.1
2025-09-07T07:36:04.4055363Z  * [new tag]                 v2.2.1-rc1                  -> v2.2.1-rc1
2025-09-07T07:36:04.4056130Z  * [new tag]                 v2.2.1-rc2                  -> v2.2.1-rc2
2025-09-07T07:36:04.4056855Z  * [new tag]                 v2.2.1-rc3                  -> v2.2.1-rc3
2025-09-07T07:36:04.4057676Z  * [new tag]                 v2.2.2                      -> v2.2.2
2025-09-07T07:36:04.4058742Z  * [new tag]                 v2.2.2-rc1                  -> v2.2.2-rc1
2025-09-07T07:36:04.4059531Z  * [new tag]                 v2.2.2-rc2                  -> v2.2.2-rc2
2025-09-07T07:36:04.4060331Z  * [new tag]                 v2.2.2-rc3                  -> v2.2.2-rc3
2025-09-07T07:36:04.4061208Z  * [new tag]                 v2.3.0                      -> v2.3.0
2025-09-07T07:36:04.4062247Z  * [new tag]                 v2.3.0-rc1                  -> v2.3.0-rc1
2025-09-07T07:36:04.4063286Z  * [new tag]                 v2.3.0-rc10                 -> v2.3.0-rc10
2025-09-07T07:36:04.4064280Z  * [new tag]                 v2.3.0-rc11                 -> v2.3.0-rc11
2025-09-07T07:36:04.4065027Z  * [new tag]                 v2.3.0-rc12                 -> v2.3.0-rc12
2025-09-07T07:36:04.4065995Z  * [new tag]                 v2.3.0-rc2                  -> v2.3.0-rc2
2025-09-07T07:36:04.4066941Z  * [new tag]                 v2.3.0-rc3                  -> v2.3.0-rc3
2025-09-07T07:36:04.4067883Z  * [new tag]                 v2.3.0-rc4                  -> v2.3.0-rc4
2025-09-07T07:36:04.4068774Z  * [new tag]                 v2.3.0-rc5                  -> v2.3.0-rc5
2025-09-07T07:36:04.4069527Z  * [new tag]                 v2.3.0-rc6                  -> v2.3.0-rc6
2025-09-07T07:36:04.4070480Z  * [new tag]                 v2.3.0-rc7                  -> v2.3.0-rc7
2025-09-07T07:36:04.4071408Z  * [new tag]                 v2.3.0-rc8                  -> v2.3.0-rc8
2025-09-07T07:36:04.4072159Z  * [new tag]                 v2.3.0-rc9                  -> v2.3.0-rc9
2025-09-07T07:36:04.4073029Z  * [new tag]                 v2.3.1                      -> v2.3.1
2025-09-07T07:36:04.4073864Z  * [new tag]                 v2.3.1-rc1                  -> v2.3.1-rc1
2025-09-07T07:36:04.4074780Z  * [new tag]                 v2.3.1-rc2                  -> v2.3.1-rc2
2025-09-07T07:36:04.4075740Z  * [new tag]                 v2.3.1-rc3                  -> v2.3.1-rc3
2025-09-07T07:36:04.4076679Z  * [new tag]                 v2.4.0                      -> v2.4.0
2025-09-07T07:36:04.4077630Z  * [new tag]                 v2.4.0-rc1                  -> v2.4.0-rc1
2025-09-07T07:36:04.4078556Z  * [new tag]                 v2.4.0-rc2                  -> v2.4.0-rc2
2025-09-07T07:36:04.4079472Z  * [new tag]                 v2.4.0-rc3                  -> v2.4.0-rc3
2025-09-07T07:36:04.4080355Z  * [new tag]                 v2.4.0-rc4                  -> v2.4.0-rc4
2025-09-07T07:36:04.4081365Z  * [new tag]                 v2.4.0-rc5                  -> v2.4.0-rc5
2025-09-07T07:36:04.4082286Z  * [new tag]                 v2.4.0-rc6                  -> v2.4.0-rc6
2025-09-07T07:36:04.4083260Z  * [new tag]                 v2.4.0-rc7                  -> v2.4.0-rc7
2025-09-07T07:36:04.4084263Z  * [new tag]                 v2.4.0-rc8                  -> v2.4.0-rc8
2025-09-07T07:36:04.4085225Z  * [new tag]                 v2.4.0-rc9                  -> v2.4.0-rc9
2025-09-07T07:36:04.4085994Z  * [new tag]                 v2.4.1                      -> v2.4.1
2025-09-07T07:36:04.4087020Z  * [new tag]                 v2.4.1-rc1                  -> v2.4.1-rc1
2025-09-07T07:36:04.4087966Z  * [new tag]                 v2.4.1-rc2                  -> v2.4.1-rc2
2025-09-07T07:36:04.4089008Z  * [new tag]                 v2.4.1-rc3                  -> v2.4.1-rc3
2025-09-07T07:36:04.4089931Z  * [new tag]                 v2.5.0                      -> v2.5.0
2025-09-07T07:36:04.4090844Z  * [new tag]                 v2.5.0-rc1                  -> v2.5.0-rc1
2025-09-07T07:36:04.4091599Z  * [new tag]                 v2.5.0-rc10                 -> v2.5.0-rc10
2025-09-07T07:36:04.4092556Z  * [new tag]                 v2.5.0-rc2                  -> v2.5.0-rc2
2025-09-07T07:36:04.4093458Z  * [new tag]                 v2.5.0-rc3                  -> v2.5.0-rc3
2025-09-07T07:36:04.4094384Z  * [new tag]                 v2.5.0-rc4                  -> v2.5.0-rc4
2025-09-07T07:36:04.4095321Z  * [new tag]                 v2.5.0-rc5                  -> v2.5.0-rc5
2025-09-07T07:36:04.4096308Z  * [new tag]                 v2.5.0-rc6                  -> v2.5.0-rc6
2025-09-07T07:36:04.4097279Z  * [new tag]                 v2.5.0-rc7                  -> v2.5.0-rc7
2025-09-07T07:36:04.4098327Z  * [new tag]                 v2.5.0-rc8                  -> v2.5.0-rc8
2025-09-07T07:36:04.4099294Z  * [new tag]                 v2.5.0-rc9                  -> v2.5.0-rc9
2025-09-07T07:36:04.4100044Z  * [new tag]                 v2.5.1                      -> v2.5.1
2025-09-07T07:36:04.4100807Z  * [new tag]                 v2.5.1-rc1                  -> v2.5.1-rc1
2025-09-07T07:36:04.4101554Z  * [new tag]                 v2.6.0                      -> v2.6.0
2025-09-07T07:36:04.4102572Z  * [new tag]                 v2.6.0-rc1                  -> v2.6.0-rc1
2025-09-07T07:36:04.4103845Z  * [new tag]                 v2.6.0-rc2                  -> v2.6.0-rc2
2025-09-07T07:36:04.4104841Z  * [new tag]                 v2.6.0-rc3                  -> v2.6.0-rc3
2025-09-07T07:36:04.4105743Z  * [new tag]                 v2.6.0-rc4                  -> v2.6.0-rc4
2025-09-07T07:36:04.4106976Z  * [new tag]                 v2.6.0-rc5                  -> v2.6.0-rc5
2025-09-07T07:36:04.4108022Z  * [new tag]                 v2.6.0-rc6                  -> v2.6.0-rc6
2025-09-07T07:36:04.4109108Z  * [new tag]                 v2.6.0-rc7                  -> v2.6.0-rc7
2025-09-07T07:36:04.4110132Z  * [new tag]                 v2.6.0-rc8                  -> v2.6.0-rc8
2025-09-07T07:36:04.4111111Z  * [new tag]                 v2.6.0-rc9                  -> v2.6.0-rc9
2025-09-07T07:36:04.4112246Z  * [new tag]                 v2.7.0                      -> v2.7.0
2025-09-07T07:36:04.4113496Z  * [new tag]                 v2.7.0-rc1                  -> v2.7.0-rc1
2025-09-07T07:36:04.4114287Z  * [new tag]                 v2.7.0-rc10                 -> v2.7.0-rc10
2025-09-07T07:36:04.4115363Z  * [new tag]                 v2.7.0-rc2                  -> v2.7.0-rc2
2025-09-07T07:36:04.4116361Z  * [new tag]                 v2.7.0-rc3                  -> v2.7.0-rc3
2025-09-07T07:36:04.4117334Z  * [new tag]                 v2.7.0-rc4                  -> v2.7.0-rc4
2025-09-07T07:36:04.4118291Z  * [new tag]                 v2.7.0-rc5                  -> v2.7.0-rc5
2025-09-07T07:36:04.4119236Z  * [new tag]                 v2.7.0-rc6                  -> v2.7.0-rc6
2025-09-07T07:36:04.4120176Z  * [new tag]                 v2.7.0-rc7                  -> v2.7.0-rc7
2025-09-07T07:36:04.4121168Z  * [new tag]                 v2.7.0-rc8                  -> v2.7.0-rc8
2025-09-07T07:36:04.4122151Z  * [new tag]                 v2.7.0-rc9                  -> v2.7.0-rc9
2025-09-07T07:36:04.4122920Z  * [new tag]                 v2.7.1                      -> v2.7.1
2025-09-07T07:36:04.4124002Z  * [new tag]                 v2.7.1-rc1                  -> v2.7.1-rc1
2025-09-07T07:36:04.4124999Z  * [new tag]                 v2.7.1-rc2                  -> v2.7.1-rc2
2025-09-07T07:36:04.4125993Z  * [new tag]                 v2.7.1-rc3                  -> v2.7.1-rc3
2025-09-07T07:36:04.4127041Z  * [new tag]                 v2.7.1-rc4                  -> v2.7.1-rc4
2025-09-07T07:36:04.4128026Z  * [new tag]                 v2.7.1-rc5                  -> v2.7.1-rc5
2025-09-07T07:36:04.4128861Z  * [new tag]                 v2.8.0                      -> v2.8.0
2025-09-07T07:36:04.4129838Z  * [new tag]                 v2.8.0-rc1                  -> v2.8.0-rc1
2025-09-07T07:36:04.4130855Z  * [new tag]                 v2.8.0-rc2                  -> v2.8.0-rc2
2025-09-07T07:36:04.4131905Z  * [new tag]                 v2.8.0-rc3                  -> v2.8.0-rc3
2025-09-07T07:36:04.4132876Z  * [new tag]                 v2.8.0-rc4                  -> v2.8.0-rc4
2025-09-07T07:36:04.4133882Z  * [new tag]                 v2.8.0-rc5                  -> v2.8.0-rc5
2025-09-07T07:36:04.4134876Z  * [new tag]                 v2.8.0-rc6                  -> v2.8.0-rc6
2025-09-07T07:36:04.4135842Z  * [new tag]                 v2.8.0-rc7                  -> v2.8.0-rc7
2025-09-07T07:36:04.4136819Z  * [new tag]                 v2.8.0-rc8                  -> v2.8.0-rc8
2025-09-07T07:36:04.4137910Z  * [new tag]                 whc_flight_1                -> whc_flight_1
2025-09-07T07:36:04.4138936Z  * [new tag]                 whc_flight_2                -> whc_flight_2
2025-09-07T07:36:04.4139818Z  * [new tag]                 whc_flight_4                -> whc_flight_4
2025-09-07T07:36:04.4965532Z [command]/usr/bin/git rev-parse --verify --quiet 93fb23d6fae7c4e82c4239a1033e522088742634^{object}
2025-09-07T07:36:04.4996750Z 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:36:04.5001076Z ##[endgroup]
2025-09-07T07:36:04.5001272Z ##[group]Determining the checkout info
2025-09-07T07:36:04.5002299Z ##[endgroup]
2025-09-07T07:36:04.5006655Z [command]/usr/bin/git sparse-checkout disable
2025-09-07T07:36:04.5049614Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-09-07T07:36:04.5080758Z ##[group]Checking out the ref
2025-09-07T07:36:04.5083906Z [command]/usr/bin/git checkout --progress --force 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:36:05.5372519Z Updating files:  85% (16573/19405)
2025-09-07T07:36:05.5501129Z Updating files:  86% (16689/19405)
2025-09-07T07:36:05.5632355Z Updating files:  87% (16883/19405)
2025-09-07T07:36:05.5734103Z Updating files:  88% (17077/19405)
2025-09-07T07:36:05.5863971Z Updating files:  89% (17271/19405)
2025-09-07T07:36:05.6021489Z Updating files:  90% (17465/19405)
2025-09-07T07:36:05.6132104Z Updating files:  91% (17659/19405)
2025-09-07T07:36:05.6263844Z Updating files:  92% (17853/19405)
2025-09-07T07:36:05.6434058Z Updating files:  93% (18047/19405)
2025-09-07T07:36:05.6616232Z Updating files:  94% (18241/19405)
2025-09-07T07:36:05.6759065Z Updating files:  95% (18435/19405)
2025-09-07T07:36:05.6907061Z Updating files:  96% (18629/19405)
2025-09-07T07:36:05.7068048Z Updating files:  97% (18823/19405)
2025-09-07T07:36:05.7294664Z Updating files:  98% (19017/19405)
2025-09-07T07:36:05.7440003Z Updating files:  99% (19211/19405)
2025-09-07T07:36:05.7440455Z Updating files: 100% (19405/19405)
2025-09-07T07:36:05.7440699Z Updating files: 100% (19405/19405), done.
2025-09-07T07:36:05.7645953Z Note: switching to '93fb23d6fae7c4e82c4239a1033e522088742634'.
2025-09-07T07:36:05.7646191Z 
2025-09-07T07:36:05.7646337Z You are in 'detached HEAD' state. You can look around, make experimental
2025-09-07T07:36:05.7646693Z changes and commit them, and you can discard any commits you make in this
2025-09-07T07:36:05.7647030Z state without impacting any branches by switching back to a branch.
2025-09-07T07:36:05.7647232Z 
2025-09-07T07:36:05.7647367Z If you want to create a new branch to retain commits you create, you may
2025-09-07T07:36:05.7647674Z do so (now or later) by using -c with the switch command. Example:
2025-09-07T07:36:05.7647855Z 
2025-09-07T07:36:05.7647937Z   git switch -c <new-branch-name>
2025-09-07T07:36:05.7648066Z 
2025-09-07T07:36:05.7648136Z Or undo this operation with:
2025-09-07T07:36:05.7648250Z 
2025-09-07T07:36:05.7648318Z   git switch -
2025-09-07T07:36:05.7648405Z 
2025-09-07T07:36:05.7648561Z Turn off this advice by setting config variable advice.detachedHead to false
2025-09-07T07:36:05.7648767Z 
2025-09-07T07:36:05.7648883Z HEAD is now at 93fb23d6fae Build vLLM nightly wheels (#162000)
2025-09-07T07:36:05.7756004Z ##[endgroup]
2025-09-07T07:36:05.7756303Z ##[group]Setting up auth for fetching submodules
2025-09-07T07:36:05.7761106Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-09-07T07:36:05.7808571Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-09-07T07:36:05.7839571Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-09-07T07:36:05.7871310Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-09-07T07:36:05.7900850Z ##[endgroup]
2025-09-07T07:36:05.7901107Z ##[group]Fetching submodules
2025-09-07T07:36:05.7903865Z [command]/usr/bin/git submodule sync --recursive
2025-09-07T07:36:05.8275708Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-09-07T07:36:05.8633887Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2025-09-07T07:36:05.8635784Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2025-09-07T07:36:05.8639096Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2025-09-07T07:36:05.8948289Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2025-09-07T07:36:05.8950709Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX'
2025-09-07T07:36:05.8954371Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2025-09-07T07:36:05.8957921Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2025-09-07T07:36:05.8961564Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter'
2025-09-07T07:36:05.8965332Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2025-09-07T07:36:05.8976452Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel'
2025-09-07T07:36:05.8978641Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2025-09-07T07:36:05.8982969Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2025-09-07T07:36:05.8987008Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2025-09-07T07:36:05.8991135Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2025-09-07T07:36:05.8995139Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2025-09-07T07:36:05.8999350Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention'
2025-09-07T07:36:05.9008480Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2025-09-07T07:36:05.9012617Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2025-09-07T07:36:05.9017095Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:36:05.9024064Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo'
2025-09-07T07:36:05.9028580Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2025-09-07T07:36:05.9032967Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2025-09-07T07:36:05.9043361Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2025-09-07T07:36:05.9048201Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2025-09-07T07:36:05.9052699Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai'
2025-09-07T07:36:05.9057358Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2025-09-07T07:36:05.9062216Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2025-09-07T07:36:05.9066951Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2025-09-07T07:36:05.9071961Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2025-09-07T07:36:05.9081854Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2025-09-07T07:36:05.9087208Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2025-09-07T07:36:05.9092380Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2025-09-07T07:36:05.9097673Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2025-09-07T07:36:05.9102818Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2025-09-07T07:36:05.9108295Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2025-09-07T07:36:05.9118866Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2025-09-07T07:36:05.9127003Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2025-09-07T07:36:05.9168010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2025-09-07T07:36:06.1212128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2025-09-07T07:36:06.1212624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2025-09-07T07:36:06.1213067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2025-09-07T07:36:06.1213477Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2025-09-07T07:36:06.1471927Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'...
2025-09-07T07:36:06.2587043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2025-09-07T07:36:06.2587618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2025-09-07T07:36:06.2588084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2025-09-07T07:36:06.2588522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2025-09-07T07:36:06.2588950Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2025-09-07T07:36:06.2589386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2025-09-07T07:36:06.2673228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2025-09-07T07:36:07.0376271Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2025-09-07T07:36:07.0376790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2025-09-07T07:36:07.0377532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2025-09-07T07:36:07.0378034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'...
2025-09-07T07:36:07.0378482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'...
2025-09-07T07:36:07.0378910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2025-09-07T07:36:07.0379336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2025-09-07T07:36:07.0379751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2025-09-07T07:36:07.0380153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2025-09-07T07:36:07.0380584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2025-09-07T07:36:07.0380995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2025-09-07T07:36:07.0381395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2025-09-07T07:36:07.0381823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2025-09-07T07:36:07.1377468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2025-09-07T07:36:14.8832692Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2025-09-07T07:36:14.8833213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2025-09-07T07:36:14.8833654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2025-09-07T07:36:14.8834067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2025-09-07T07:36:14.8834478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2025-09-07T07:36:14.8834948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'...
2025-09-07T07:36:14.8835843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'...
2025-09-07T07:36:14.8836280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2025-09-07T07:36:14.8836722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2025-09-07T07:36:14.8837136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2025-09-07T07:36:14.9014561Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-09-07T07:36:14.9165231Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-09-07T07:36:14.9275601Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-09-07T07:36:14.9549510Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-09-07T07:36:15.0299929Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07'
2025-09-07T07:36:15.0795061Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-09-07T07:36:15.8194568Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-09-07T07:36:15.9680946Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-09-07T07:36:15.9707876Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:36:15.9744728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'...
2025-09-07T07:36:19.0414484Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-09-07T07:36:19.0683816Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-09-07T07:36:19.3949749Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-09-07T07:36:19.4436552Z Submodule path 'third_party/cpp-httplib': checked out '89c932f313c6437c38f2982869beacc89c2f2246'
2025-09-07T07:36:19.5357502Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-09-07T07:36:19.5805042Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6'
2025-09-07T07:36:20.1757349Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491'
2025-09-07T07:36:20.3217280Z Submodule path 'third_party/fbgemm': checked out '4b39c551efe15e6bbade20565b0ceb2d8ce3352d'
2025-09-07T07:36:20.3246180Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit'
2025-09-07T07:36:20.3247937Z Submodule 'external/composable_kernel' (https://github.com/jwfromm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:36:20.3250660Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:36:20.3253589Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass'
2025-09-07T07:36:20.3256617Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest'
2025-09-07T07:36:20.3259816Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:36:20.3262721Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json'
2025-09-07T07:36:20.3303519Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'...
2025-09-07T07:36:21.2744936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'...
2025-09-07T07:36:21.2745540Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'...
2025-09-07T07:36:21.2746050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'...
2025-09-07T07:36:21.2746572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'...
2025-09-07T07:36:21.3744959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'...
2025-09-07T07:36:21.8822275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'...
2025-09-07T07:36:25.3485835Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea'
2025-09-07T07:36:25.6160304Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9'
2025-09-07T07:36:25.7097536Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-09-07T07:36:26.3068161Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '311f3c8e51dc0eb56310cfc6980bf63d0fbd7917'
2025-09-07T07:36:26.3524176Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-09-07T07:36:26.3665756Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out '63b6a7b541fa7f08f8475ca7d74054db36ff2691'
2025-09-07T07:36:26.4747191Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-09-07T07:36:26.5496732Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-09-07T07:36:26.5522556Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:36:26.5523861Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:36:26.5559574Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'...
2025-09-07T07:36:29.2995217Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'...
2025-09-07T07:36:29.5444896Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-09-07T07:36:30.0802007Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-09-07T07:36:30.2220973Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-09-07T07:36:30.2554687Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-09-07T07:36:30.2948648Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-09-07T07:36:30.3230126Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2'
2025-09-07T07:36:30.3689425Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-09-07T07:36:30.3842550Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-09-07T07:36:30.3865327Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2025-09-07T07:36:30.3898386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2025-09-07T07:36:40.7427807Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-09-07T07:36:40.7661127Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-09-07T07:36:40.8477653Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658'
2025-09-07T07:36:40.8503566Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:36:40.8505104Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:36:40.8508382Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:36:40.8546321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2025-09-07T07:36:41.4281370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2025-09-07T07:36:41.6269147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2025-09-07T07:36:41.7043976Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-09-07T07:36:41.7067611Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:36:41.7069523Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:36:41.7072531Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:36:41.7075625Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:36:41.7078812Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:36:41.7082138Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:36:41.7085315Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:36:41.7088682Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:36:41.7128817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2025-09-07T07:36:42.7865748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2025-09-07T07:36:42.7866541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2025-09-07T07:36:42.7867234Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2025-09-07T07:36:42.7867920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2025-09-07T07:36:42.7868580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2025-09-07T07:36:42.7869238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2025-09-07T07:36:42.8866448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2025-09-07T07:36:46.7563592Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-09-07T07:36:46.7775087Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-09-07T07:36:46.8139412Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-09-07T07:36:46.8296924Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-09-07T07:36:46.8317473Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:36:46.8353022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2025-09-07T07:36:47.0897895Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-09-07T07:36:47.1119296Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-09-07T07:36:47.1524585Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-09-07T07:36:47.2524248Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-09-07T07:36:47.2717967Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-09-07T07:36:47.3105772Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-09-07T07:36:47.3668530Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-09-07T07:36:47.4130983Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7'
2025-09-07T07:36:47.4592520Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-09-07T07:36:47.5694939Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-09-07T07:36:48.0213807Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-09-07T07:36:48.0255530Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2025-09-07T07:36:48.0293250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2025-09-07T07:36:48.8117629Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-09-07T07:36:48.8870876Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-09-07T07:36:48.8897171Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:36:48.8898462Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:36:48.8901314Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:36:48.8904707Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:36:48.8907728Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:36:48.8921887Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:36:48.8922786Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:36:48.8923520Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:36:48.8955629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2025-09-07T07:36:49.6881334Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2025-09-07T07:36:49.6882052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2025-09-07T07:36:49.6882685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2025-09-07T07:36:49.6883301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2025-09-07T07:36:49.7881704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2025-09-07T07:36:49.9403866Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2025-09-07T07:36:54.2464668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2025-09-07T07:36:54.8804356Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-09-07T07:36:54.9200546Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-09-07T07:36:54.9379207Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-09-07T07:36:55.0415020Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-09-07T07:36:55.0581252Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-09-07T07:36:55.0754353Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-09-07T07:36:55.0939821Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-09-07T07:36:55.0963443Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:36:55.0964871Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:36:55.1002828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2025-09-07T07:36:56.4030599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2025-09-07T07:36:56.6266068Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-09-07T07:36:56.6732316Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-09-07T07:36:57.2035844Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-09-07T07:36:57.2183705Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-09-07T07:36:57.4789694Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-09-07T07:36:57.4819483Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:36:57.4821130Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2025-09-07T07:36:57.4858973Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2025-09-07T07:36:58.5353333Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2025-09-07T07:36:58.5509954Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-09-07T07:36:58.6167108Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-09-07T07:36:58.6289600Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-09-07T07:36:58.6438607Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-09-07T07:36:58.6839615Z Submodule path 'third_party/pybind11': checked out 'f5fbe867d2d26e4a0a9177a51f6e568868ad3dc8'
2025-09-07T07:36:58.7136137Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-09-07T07:36:58.7564954Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-09-07T07:36:58.7868018Z Submodule path 'third_party/tensorpipe': checked out 'af0118d13e52f5a08841464a768e01a0bf3e3075'
2025-09-07T07:36:58.7890283Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:36:58.7891528Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:36:58.7894364Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:36:58.7897672Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:36:58.7932928Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2025-09-07T07:36:59.4952081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2025-09-07T07:36:59.5218336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2025-09-07T07:36:59.6915268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2025-09-07T07:36:59.7459386Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-09-07T07:36:59.7640405Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-09-07T07:36:59.8334125Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b'
2025-09-07T07:36:59.8643257Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-09-07T07:36:59.8664447Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:36:59.8698902Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2025-09-07T07:37:00.0686581Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-09-07T07:37:00.0733261Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-09-07T07:37:00.1094195Z Entering 'android/libs/fbjni'
2025-09-07T07:37:00.1145162Z Entering 'third_party/FP16'
2025-09-07T07:37:00.1196696Z Entering 'third_party/FXdiv'
2025-09-07T07:37:00.1248267Z Entering 'third_party/NNPACK'
2025-09-07T07:37:00.1297906Z Entering 'third_party/NVTX'
2025-09-07T07:37:00.1348548Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T07:37:00.1395876Z Entering 'third_party/XNNPACK'
2025-09-07T07:37:00.1457731Z Entering 'third_party/aiter'
2025-09-07T07:37:00.1508124Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:37:00.1562868Z Entering 'third_party/benchmark'
2025-09-07T07:37:00.1611257Z Entering 'third_party/composable_kernel'
2025-09-07T07:37:00.1669906Z Entering 'third_party/cpp-httplib'
2025-09-07T07:37:00.1719171Z Entering 'third_party/cpuinfo'
2025-09-07T07:37:00.1770494Z Entering 'third_party/cudnn_frontend'
2025-09-07T07:37:00.1820309Z Entering 'third_party/cutlass'
2025-09-07T07:37:00.1877527Z Entering 'third_party/fbgemm'
2025-09-07T07:37:00.1931461Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T07:37:00.1978490Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:37:00.2033680Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:37:00.2079745Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T07:37:00.2137574Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T07:37:00.2184251Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:37:00.2229958Z Entering 'third_party/fbgemm/external/json'
2025-09-07T07:37:00.2280955Z Entering 'third_party/flash-attention'
2025-09-07T07:37:00.2332320Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:37:00.2384352Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:37:00.2441290Z Entering 'third_party/flatbuffers'
2025-09-07T07:37:00.2493102Z Entering 'third_party/fmt'
2025-09-07T07:37:00.2543418Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:37:00.2593789Z Entering 'third_party/gloo'
2025-09-07T07:37:00.2645482Z Entering 'third_party/googletest'
2025-09-07T07:37:00.2695007Z Entering 'third_party/ideep'
2025-09-07T07:37:00.2745296Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T07:37:00.2800885Z Entering 'third_party/ittapi'
2025-09-07T07:37:00.2848997Z Entering 'third_party/kineto'
2025-09-07T07:37:00.2897925Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:37:00.2943598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:37:00.2991350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:37:00.3042750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:37:00.3089124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:37:00.3140053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:37:00.3188776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:37:00.3235141Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:37:00.3282485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:37:00.3332477Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:37:00.3383025Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:37:00.3431716Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:37:00.3484630Z Entering 'third_party/kleidiai'
2025-09-07T07:37:00.3537925Z Entering 'third_party/mimalloc'
2025-09-07T07:37:00.3590571Z Entering 'third_party/nlohmann'
2025-09-07T07:37:00.3642413Z Entering 'third_party/onnx'
2025-09-07T07:37:00.3707101Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T07:37:00.3757002Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T07:37:00.3807684Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:37:00.3853320Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:37:00.3902861Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:37:00.3948546Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:37:00.3994930Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:37:00.4041535Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:37:00.4087005Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:37:00.4135042Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:37:00.4184735Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:37:00.4234688Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:37:00.4301152Z Entering 'third_party/pocketfft'
2025-09-07T07:37:00.4352898Z Entering 'third_party/protobuf'
2025-09-07T07:37:00.4407510Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:37:00.4453626Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T07:37:00.4508991Z Entering 'third_party/psimd'
2025-09-07T07:37:00.4559105Z Entering 'third_party/pthreadpool'
2025-09-07T07:37:00.4606196Z Entering 'third_party/pybind11'
2025-09-07T07:37:00.4653411Z Entering 'third_party/python-peachpy'
2025-09-07T07:37:00.4701181Z Entering 'third_party/sleef'
2025-09-07T07:37:00.4751944Z Entering 'third_party/tensorpipe'
2025-09-07T07:37:00.4801218Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:37:00.4848804Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:37:00.4895624Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:37:00.4942335Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:37:00.4989288Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:37:00.5057373Z ##[endgroup]
2025-09-07T07:37:00.5057737Z ##[group]Persisting credentials for submodules
2025-09-07T07:37:00.5063402Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-09-07T07:37:00.5421569Z Entering 'android/libs/fbjni'
2025-09-07T07:37:00.5487191Z Entering 'third_party/FP16'
2025-09-07T07:37:00.5550982Z Entering 'third_party/FXdiv'
2025-09-07T07:37:00.5615090Z Entering 'third_party/NNPACK'
2025-09-07T07:37:00.5678714Z Entering 'third_party/NVTX'
2025-09-07T07:37:00.5743945Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T07:37:00.5807051Z Entering 'third_party/XNNPACK'
2025-09-07T07:37:00.5884759Z Entering 'third_party/aiter'
2025-09-07T07:37:00.5952934Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:37:00.6027057Z Entering 'third_party/benchmark'
2025-09-07T07:37:00.6093579Z Entering 'third_party/composable_kernel'
2025-09-07T07:37:00.6165040Z Entering 'third_party/cpp-httplib'
2025-09-07T07:37:00.6230961Z Entering 'third_party/cpuinfo'
2025-09-07T07:37:00.6296805Z Entering 'third_party/cudnn_frontend'
2025-09-07T07:37:00.6358968Z Entering 'third_party/cutlass'
2025-09-07T07:37:00.6428827Z Entering 'third_party/fbgemm'
2025-09-07T07:37:00.6493816Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T07:37:00.6557486Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:37:00.6627906Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:37:00.6691674Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T07:37:00.6764931Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T07:37:00.6832651Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:37:00.6897833Z Entering 'third_party/fbgemm/external/json'
2025-09-07T07:37:00.6967036Z Entering 'third_party/flash-attention'
2025-09-07T07:37:00.7033262Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:37:00.7101623Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:37:00.7176529Z Entering 'third_party/flatbuffers'
2025-09-07T07:37:00.7243534Z Entering 'third_party/fmt'
2025-09-07T07:37:00.7307239Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:37:00.7370190Z Entering 'third_party/gloo'
2025-09-07T07:37:00.7434331Z Entering 'third_party/googletest'
2025-09-07T07:37:00.7498548Z Entering 'third_party/ideep'
2025-09-07T07:37:00.7563266Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T07:37:00.7636871Z Entering 'third_party/ittapi'
2025-09-07T07:37:00.7703664Z Entering 'third_party/kineto'
2025-09-07T07:37:00.7770763Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:37:00.7834102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:37:00.7902520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:37:00.7968005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:37:00.8029848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:37:00.8095647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:37:00.8159619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:37:00.8222971Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:37:00.8285881Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:37:00.8353007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:37:00.8420614Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:37:00.8486959Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:37:00.8556034Z Entering 'third_party/kleidiai'
2025-09-07T07:37:00.8625297Z Entering 'third_party/mimalloc'
2025-09-07T07:37:00.8693464Z Entering 'third_party/nlohmann'
2025-09-07T07:37:00.8760256Z Entering 'third_party/onnx'
2025-09-07T07:37:00.8837068Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T07:37:00.8901871Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T07:37:00.8966031Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:37:00.9026414Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:37:00.9089692Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:37:00.9151199Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:37:00.9215667Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:37:00.9276524Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:37:00.9342137Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:37:00.9406532Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:37:00.9475471Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:37:00.9543600Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:37:00.9622974Z Entering 'third_party/pocketfft'
2025-09-07T07:37:00.9685664Z Entering 'third_party/protobuf'
2025-09-07T07:37:00.9750544Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:37:00.9811996Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T07:37:00.9878196Z Entering 'third_party/psimd'
2025-09-07T07:37:00.9940797Z Entering 'third_party/pthreadpool'
2025-09-07T07:37:01.0005112Z Entering 'third_party/pybind11'
2025-09-07T07:37:01.0071402Z Entering 'third_party/python-peachpy'
2025-09-07T07:37:01.0140067Z Entering 'third_party/sleef'
2025-09-07T07:37:01.0206474Z Entering 'third_party/tensorpipe'
2025-09-07T07:37:01.0272927Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:37:01.0338752Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:37:01.0402105Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:37:01.0466362Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:37:01.0530281Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:37:01.0616146Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-09-07T07:37:01.0961883Z Entering 'android/libs/fbjni'
2025-09-07T07:37:01.1024752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-09-07T07:37:01.1045403Z Entering 'third_party/FP16'
2025-09-07T07:37:01.1108832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-09-07T07:37:01.1128076Z Entering 'third_party/FXdiv'
2025-09-07T07:37:01.1187482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-09-07T07:37:01.1207658Z Entering 'third_party/NNPACK'
2025-09-07T07:37:01.1269993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-09-07T07:37:01.1288936Z Entering 'third_party/NVTX'
2025-09-07T07:37:01.1350934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-09-07T07:37:01.1369894Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T07:37:01.1428032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-09-07T07:37:01.1447161Z Entering 'third_party/XNNPACK'
2025-09-07T07:37:01.1504821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-09-07T07:37:01.1535847Z Entering 'third_party/aiter'
2025-09-07T07:37:01.1594613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-09-07T07:37:01.1615835Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:37:01.1673017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-09-07T07:37:01.1701889Z Entering 'third_party/benchmark'
2025-09-07T07:37:01.1763943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-09-07T07:37:01.1783224Z Entering 'third_party/composable_kernel'
2025-09-07T07:37:01.1846008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-09-07T07:37:01.1872314Z Entering 'third_party/cpp-httplib'
2025-09-07T07:37:01.1932821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-09-07T07:37:01.1952909Z Entering 'third_party/cpuinfo'
2025-09-07T07:37:01.2016718Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-09-07T07:37:01.2035833Z Entering 'third_party/cudnn_frontend'
2025-09-07T07:37:01.2096739Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-09-07T07:37:01.2116401Z Entering 'third_party/cutlass'
2025-09-07T07:37:01.2172686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-09-07T07:37:01.2199583Z Entering 'third_party/fbgemm'
2025-09-07T07:37:01.2257719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-09-07T07:37:01.2279431Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T07:37:01.2339432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-09-07T07:37:01.2356840Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:37:01.2415174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-09-07T07:37:01.2438927Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:37:01.2499411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-09-07T07:37:01.2521713Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T07:37:01.2581105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-09-07T07:37:01.2608376Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T07:37:01.2668447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-09-07T07:37:01.2688072Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:37:01.2747907Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-09-07T07:37:01.2766702Z Entering 'third_party/fbgemm/external/json'
2025-09-07T07:37:01.2827598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-09-07T07:37:01.2851445Z Entering 'third_party/flash-attention'
2025-09-07T07:37:01.2910825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-09-07T07:37:01.2931411Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:37:01.2989323Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-09-07T07:37:01.3013771Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:37:01.3073409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-09-07T07:37:01.3100926Z Entering 'third_party/flatbuffers'
2025-09-07T07:37:01.3158372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-09-07T07:37:01.3180363Z Entering 'third_party/fmt'
2025-09-07T07:37:01.3237918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-09-07T07:37:01.3258682Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:37:01.3317543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-09-07T07:37:01.3337645Z Entering 'third_party/gloo'
2025-09-07T07:37:01.3396400Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-09-07T07:37:01.3417892Z Entering 'third_party/googletest'
2025-09-07T07:37:01.3477797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-09-07T07:37:01.3499146Z Entering 'third_party/ideep'
2025-09-07T07:37:01.3560335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-09-07T07:37:01.3579873Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T07:37:01.3639827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-09-07T07:37:01.3667625Z Entering 'third_party/ittapi'
2025-09-07T07:37:01.3729001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-09-07T07:37:01.3748837Z Entering 'third_party/kineto'
2025-09-07T07:37:01.3810506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-09-07T07:37:01.3831008Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:37:01.3890080Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-09-07T07:37:01.3906820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:37:01.3964122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-09-07T07:37:01.3984471Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:37:01.4042794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-09-07T07:37:01.4061845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:37:01.4122695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-09-07T07:37:01.4141651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:37:01.4200437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-09-07T07:37:01.4219461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:37:01.4280475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-09-07T07:37:01.4302195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:37:01.4361448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-09-07T07:37:01.4382115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:37:01.4442961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-09-07T07:37:01.4463856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:37:01.4526600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-09-07T07:37:01.4546121Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:37:01.4607554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-09-07T07:37:01.4630166Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:37:01.4687291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-09-07T07:37:01.4706416Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:37:01.4762513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-09-07T07:37:01.4784083Z Entering 'third_party/kleidiai'
2025-09-07T07:37:01.4846116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-09-07T07:37:01.4866283Z Entering 'third_party/mimalloc'
2025-09-07T07:37:01.4927095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-09-07T07:37:01.4946068Z Entering 'third_party/nlohmann'
2025-09-07T07:37:01.5004303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-09-07T07:37:01.5025311Z Entering 'third_party/onnx'
2025-09-07T07:37:01.5087991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-09-07T07:37:01.5121372Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T07:37:01.5179954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-09-07T07:37:01.5204374Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T07:37:01.5265632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-09-07T07:37:01.5288109Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:37:01.5349335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-09-07T07:37:01.5369562Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:37:01.5430927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-09-07T07:37:01.5449421Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:37:01.5510931Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-09-07T07:37:01.5529365Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:37:01.5586749Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-09-07T07:37:01.5609250Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:37:01.5666314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-09-07T07:37:01.5684280Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:37:01.5741514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-09-07T07:37:01.5759251Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:37:01.5816837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-09-07T07:37:01.5834549Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:37:01.5892164Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-09-07T07:37:01.5913698Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:37:01.5974394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-09-07T07:37:01.5997750Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:37:01.6059745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-09-07T07:37:01.6096576Z Entering 'third_party/pocketfft'
2025-09-07T07:37:01.6159555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-09-07T07:37:01.6179722Z Entering 'third_party/protobuf'
2025-09-07T07:37:01.6241142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-09-07T07:37:01.6264611Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:37:01.6321869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-09-07T07:37:01.6342332Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T07:37:01.6403652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-09-07T07:37:01.6425126Z Entering 'third_party/psimd'
2025-09-07T07:37:01.6483752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-09-07T07:37:01.6502851Z Entering 'third_party/pthreadpool'
2025-09-07T07:37:01.6562249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-09-07T07:37:01.6580663Z Entering 'third_party/pybind11'
2025-09-07T07:37:01.6638756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-09-07T07:37:01.6659368Z Entering 'third_party/python-peachpy'
2025-09-07T07:37:01.6719770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-09-07T07:37:01.6740556Z Entering 'third_party/sleef'
2025-09-07T07:37:01.6800065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-09-07T07:37:01.6821135Z Entering 'third_party/tensorpipe'
2025-09-07T07:37:01.6881901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-09-07T07:37:01.6901423Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:37:01.6962723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-09-07T07:37:01.6984155Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:37:01.7045117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-09-07T07:37:01.7062404Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:37:01.7121776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-09-07T07:37:01.7140384Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:37:01.7198265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-09-07T07:37:01.7219934Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:37:01.7277400Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-09-07T07:37:01.7840097Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-09-07T07:37:01.8201405Z Entering 'android/libs/fbjni'
2025-09-07T07:37:01.8250732Z Entering 'third_party/FP16'
2025-09-07T07:37:01.8302046Z Entering 'third_party/FXdiv'
2025-09-07T07:37:01.8349947Z Entering 'third_party/NNPACK'
2025-09-07T07:37:01.8400664Z Entering 'third_party/NVTX'
2025-09-07T07:37:01.8452319Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T07:37:01.8500793Z Entering 'third_party/XNNPACK'
2025-09-07T07:37:01.8565234Z Entering 'third_party/aiter'
2025-09-07T07:37:01.8617589Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:37:01.8673796Z Entering 'third_party/benchmark'
2025-09-07T07:37:01.8724747Z Entering 'third_party/composable_kernel'
2025-09-07T07:37:01.8783617Z Entering 'third_party/cpp-httplib'
2025-09-07T07:37:01.8833018Z Entering 'third_party/cpuinfo'
2025-09-07T07:37:01.8880893Z Entering 'third_party/cudnn_frontend'
2025-09-07T07:37:01.8928272Z Entering 'third_party/cutlass'
2025-09-07T07:37:01.8983058Z Entering 'third_party/fbgemm'
2025-09-07T07:37:01.9036406Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T07:37:01.9081848Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:37:01.9134735Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:37:01.9181699Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T07:37:01.9238642Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T07:37:01.9289436Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:37:01.9340576Z Entering 'third_party/fbgemm/external/json'
2025-09-07T07:37:01.9394568Z Entering 'third_party/flash-attention'
2025-09-07T07:37:01.9447120Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:37:01.9500293Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:37:01.9560401Z Entering 'third_party/flatbuffers'
2025-09-07T07:37:01.9612125Z Entering 'third_party/fmt'
2025-09-07T07:37:01.9658881Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:37:01.9707532Z Entering 'third_party/gloo'
2025-09-07T07:37:01.9755129Z Entering 'third_party/googletest'
2025-09-07T07:37:01.9804608Z Entering 'third_party/ideep'
2025-09-07T07:37:01.9852023Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T07:37:01.9905578Z Entering 'third_party/ittapi'
2025-09-07T07:37:01.9952579Z Entering 'third_party/kineto'
2025-09-07T07:37:02.0001017Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:37:02.0049470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:37:02.0097786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:37:02.0148478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:37:02.0196082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:37:02.0247036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:37:02.0297917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:37:02.0348475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:37:02.0398975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:37:02.0452662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:37:02.0504951Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:37:02.0551531Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:37:02.0603265Z Entering 'third_party/kleidiai'
2025-09-07T07:37:02.0653918Z Entering 'third_party/mimalloc'
2025-09-07T07:37:02.0701454Z Entering 'third_party/nlohmann'
2025-09-07T07:37:02.0751935Z Entering 'third_party/onnx'
2025-09-07T07:37:02.0812961Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T07:37:02.0864295Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T07:37:02.0915969Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:37:02.0963999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:37:02.1012413Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:37:02.1061084Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:37:02.1111061Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:37:02.1160012Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:37:02.1210483Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:37:02.1260292Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:37:02.1307912Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:37:02.1358663Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:37:02.1422087Z Entering 'third_party/pocketfft'
2025-09-07T07:37:02.1468967Z Entering 'third_party/protobuf'
2025-09-07T07:37:02.1519314Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:37:02.1564867Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T07:37:02.1615356Z Entering 'third_party/psimd'
2025-09-07T07:37:02.1661986Z Entering 'third_party/pthreadpool'
2025-09-07T07:37:02.1711604Z Entering 'third_party/pybind11'
2025-09-07T07:37:02.1762409Z Entering 'third_party/python-peachpy'
2025-09-07T07:37:02.1812762Z Entering 'third_party/sleef'
2025-09-07T07:37:02.1862540Z Entering 'third_party/tensorpipe'
2025-09-07T07:37:02.1912891Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:37:02.1960820Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:37:02.2011387Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:37:02.2059187Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:37:02.2105472Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:37:02.2179349Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-09-07T07:37:02.2527375Z Entering 'android/libs/fbjni'
2025-09-07T07:37:02.2576880Z Entering 'third_party/FP16'
2025-09-07T07:37:02.2628467Z Entering 'third_party/FXdiv'
2025-09-07T07:37:02.2678981Z Entering 'third_party/NNPACK'
2025-09-07T07:37:02.2731066Z Entering 'third_party/NVTX'
2025-09-07T07:37:02.2783654Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T07:37:02.2836490Z Entering 'third_party/XNNPACK'
2025-09-07T07:37:02.2898778Z Entering 'third_party/aiter'
2025-09-07T07:37:02.2949897Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:37:02.3009016Z Entering 'third_party/benchmark'
2025-09-07T07:37:02.3056801Z Entering 'third_party/composable_kernel'
2025-09-07T07:37:02.3113499Z Entering 'third_party/cpp-httplib'
2025-09-07T07:37:02.3162140Z Entering 'third_party/cpuinfo'
2025-09-07T07:37:02.3217118Z Entering 'third_party/cudnn_frontend'
2025-09-07T07:37:02.3264762Z Entering 'third_party/cutlass'
2025-09-07T07:37:02.3321133Z Entering 'third_party/fbgemm'
2025-09-07T07:37:02.3371823Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T07:37:02.3418937Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:37:02.3472989Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:37:02.3522078Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T07:37:02.3578772Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T07:37:02.3629381Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:37:02.3677890Z Entering 'third_party/fbgemm/external/json'
2025-09-07T07:37:02.3730226Z Entering 'third_party/flash-attention'
2025-09-07T07:37:02.3781480Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:37:02.3836828Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:37:02.3892095Z Entering 'third_party/flatbuffers'
2025-09-07T07:37:02.3944798Z Entering 'third_party/fmt'
2025-09-07T07:37:02.3991439Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:37:02.4039523Z Entering 'third_party/gloo'
2025-09-07T07:37:02.4086646Z Entering 'third_party/googletest'
2025-09-07T07:37:02.4133867Z Entering 'third_party/ideep'
2025-09-07T07:37:02.4181933Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T07:37:02.4234635Z Entering 'third_party/ittapi'
2025-09-07T07:37:02.4281874Z Entering 'third_party/kineto'
2025-09-07T07:37:02.4335513Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:37:02.4382895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:37:02.4434607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:37:02.4485019Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:37:02.4534819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:37:02.4583485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:37:02.4640470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:37:02.4688468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:37:02.4740231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:37:02.4787343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:37:02.4837784Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:37:02.4884510Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:37:02.4934465Z Entering 'third_party/kleidiai'
2025-09-07T07:37:02.4982924Z Entering 'third_party/mimalloc'
2025-09-07T07:37:02.5032868Z Entering 'third_party/nlohmann'
2025-09-07T07:37:02.5085333Z Entering 'third_party/onnx'
2025-09-07T07:37:02.5147045Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T07:37:02.5200197Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T07:37:02.5254477Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:37:02.5301071Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:37:02.5352640Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:37:02.5402763Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:37:02.5458088Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:37:02.5505389Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:37:02.5551863Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:37:02.5597503Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:37:02.5649006Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:37:02.5699283Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:37:02.5763497Z Entering 'third_party/pocketfft'
2025-09-07T07:37:02.5811849Z Entering 'third_party/protobuf'
2025-09-07T07:37:02.5863602Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:37:02.5910306Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T07:37:02.5964178Z Entering 'third_party/psimd'
2025-09-07T07:37:02.6015450Z Entering 'third_party/pthreadpool'
2025-09-07T07:37:02.6064690Z Entering 'third_party/pybind11'
2025-09-07T07:37:02.6115436Z Entering 'third_party/python-peachpy'
2025-09-07T07:37:02.6165155Z Entering 'third_party/sleef'
2025-09-07T07:37:02.6216608Z Entering 'third_party/tensorpipe'
2025-09-07T07:37:02.6267595Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:37:02.6314480Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:37:02.6362150Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:37:02.6412093Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:37:02.6457598Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:37:02.6522788Z ##[endgroup]
2025-09-07T07:37:02.6562577Z [command]/usr/bin/git log -1 --format=%H
2025-09-07T07:37:02.6588594Z 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:37:02.6688800Z ##[group]Run cd "${GITHUB_WORKSPACE}"
2025-09-07T07:37:02.6689049Z cd "${GITHUB_WORKSPACE}"
2025-09-07T07:37:02.6689248Z # Clean stale submodule dirs
2025-09-07T07:37:02.6689451Z if [ -z "${NO_SUDO}" ]; then
2025-09-07T07:37:02.6689701Z   sudo git submodule foreach --recursive git clean -ffdx
2025-09-07T07:37:02.6689943Z else
2025-09-07T07:37:02.6690283Z   git submodule foreach --recursive git clean -ffdx
2025-09-07T07:37:02.6690510Z fi
2025-09-07T07:37:02.6702610Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:02.6702846Z env:
2025-09-07T07:37:02.6702990Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:02.6703373Z   NO_SUDO: true
2025-09-07T07:37:02.6703519Z ##[endgroup]
2025-09-07T07:37:02.7092771Z Entering 'android/libs/fbjni'
2025-09-07T07:37:02.7134182Z Entering 'third_party/FP16'
2025-09-07T07:37:02.7171873Z Entering 'third_party/FXdiv'
2025-09-07T07:37:02.7209165Z Entering 'third_party/NNPACK'
2025-09-07T07:37:02.7253432Z Entering 'third_party/NVTX'
2025-09-07T07:37:02.7295786Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T07:37:02.7332562Z Entering 'third_party/XNNPACK'
2025-09-07T07:37:02.7452952Z Entering 'third_party/aiter'
2025-09-07T07:37:02.7499881Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T07:37:02.7619585Z Entering 'third_party/benchmark'
2025-09-07T07:37:02.7659341Z Entering 'third_party/composable_kernel'
2025-09-07T07:37:02.7787516Z Entering 'third_party/cpp-httplib'
2025-09-07T07:37:02.7826591Z Entering 'third_party/cpuinfo'
2025-09-07T07:37:02.7871231Z Entering 'third_party/cudnn_frontend'
2025-09-07T07:37:02.7915548Z Entering 'third_party/cutlass'
2025-09-07T07:37:02.8017576Z Entering 'third_party/fbgemm'
2025-09-07T07:37:02.8085058Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T07:37:02.8121649Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T07:37:02.8236502Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T07:37:02.8274661Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T07:37:02.8377367Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T07:37:02.8416082Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T07:37:02.8449543Z Entering 'third_party/fbgemm/external/json'
2025-09-07T07:37:02.8500617Z Entering 'third_party/flash-attention'
2025-09-07T07:37:02.8547625Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T07:37:02.8655594Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T07:37:02.8752140Z Entering 'third_party/flatbuffers'
2025-09-07T07:37:02.8832988Z Entering 'third_party/fmt'
2025-09-07T07:37:02.8870985Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T07:37:02.8913240Z Entering 'third_party/gloo'
2025-09-07T07:37:02.8950693Z Entering 'third_party/googletest'
2025-09-07T07:37:02.8995742Z Entering 'third_party/ideep'
2025-09-07T07:37:02.9034318Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T07:37:02.9122863Z Entering 'third_party/ittapi'
2025-09-07T07:37:02.9161237Z Entering 'third_party/kineto'
2025-09-07T07:37:02.9201258Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T07:37:02.9240595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T07:37:02.9291134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T07:37:02.9328290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T07:37:02.9366399Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T07:37:02.9402663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T07:37:02.9444193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T07:37:02.9481449Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T07:37:02.9524709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T07:37:02.9571596Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T07:37:02.9613220Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T07:37:02.9651117Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T07:37:02.9692632Z Entering 'third_party/kleidiai'
2025-09-07T07:37:02.9734800Z Entering 'third_party/mimalloc'
2025-09-07T07:37:02.9774662Z Entering 'third_party/nlohmann'
2025-09-07T07:37:02.9828845Z Entering 'third_party/onnx'
2025-09-07T07:37:03.0190238Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T07:37:03.0237755Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T07:37:03.0304709Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T07:37:03.0340284Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T07:37:03.0382782Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T07:37:03.0420188Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T07:37:03.0469910Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T07:37:03.0505318Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T07:37:03.0541034Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T07:37:03.0580089Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T07:37:03.0635199Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T07:37:03.0674501Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T07:37:03.0953602Z Entering 'third_party/pocketfft'
2025-09-07T07:37:03.0992519Z Entering 'third_party/protobuf'
2025-09-07T07:37:03.1079456Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T07:37:03.1118077Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T07:37:03.1162121Z Entering 'third_party/psimd'
2025-09-07T07:37:03.1202230Z Entering 'third_party/pthreadpool'
2025-09-07T07:37:03.1239626Z Entering 'third_party/pybind11'
2025-09-07T07:37:03.1280300Z Entering 'third_party/python-peachpy'
2025-09-07T07:37:03.1321355Z Entering 'third_party/sleef'
2025-09-07T07:37:03.1361262Z Entering 'third_party/tensorpipe'
2025-09-07T07:37:03.1400641Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T07:37:03.1440592Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T07:37:03.1474276Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T07:37:03.1517208Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T07:37:03.1551764Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T07:37:03.1686402Z Prepare all required actions
2025-09-07T07:37:03.1686836Z Getting action download info
2025-09-07T07:37:03.2795578Z ##[group]Run ./.github/actions/setup-linux
2025-09-07T07:37:03.2795798Z env:
2025-09-07T07:37:03.2795955Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:03.2796122Z ##[endgroup]
2025-09-07T07:37:03.2826944Z ##[group]Run set -euo pipefail
2025-09-07T07:37:03.2827177Z set -euo pipefail
2025-09-07T07:37:03.2827363Z function get_ec2_metadata() {
2025-09-07T07:37:03.2827609Z   # Pulled from instance metadata endpoint for EC2
2025-09-07T07:37:03.2828021Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2025-09-07T07:37:03.2828369Z   category=$1
2025-09-07T07:37:03.2828602Z   # If it is GCP runner (runner name contains gcp), do not run this
2025-09-07T07:37:03.2828873Z   runner_name_str=i-0d9447236daff7b96
2025-09-07T07:37:03.2829109Z   if [[ -f /.inarc ]]; then
2025-09-07T07:37:03.2829330Z     echo "ARC Runner, no info on ec2 metadata"
2025-09-07T07:37:03.2829571Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2025-09-07T07:37:03.2829855Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2025-09-07T07:37:03.2830113Z   else
2025-09-07T07:37:03.2830635Z     curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2025-09-07T07:37:03.2831160Z   fi
2025-09-07T07:37:03.2831301Z }
2025-09-07T07:37:03.2831482Z echo "ami-id: $(get_ec2_metadata ami-id)"
2025-09-07T07:37:03.2831745Z echo "instance-id: $(get_ec2_metadata instance-id)"
2025-09-07T07:37:03.2832039Z echo "instance-type: $(get_ec2_metadata instance-type)"
2025-09-07T07:37:03.2832289Z echo "system info $(uname -a)"
2025-09-07T07:37:03.2840522Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:03.2840761Z env:
2025-09-07T07:37:03.2840908Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:03.2841075Z ##[endgroup]
2025-09-07T07:37:03.2999929Z ami-id: ami-05ffe3c48a9991133
2025-09-07T07:37:03.3115204Z instance-id: i-0d9447236daff7b96
2025-09-07T07:37:03.3220431Z instance-type: m7a.24xlarge
2025-09-07T07:37:03.3233005Z system info Linux ip-10-0-57-198.ec2.internal 6.1.141-155.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 17 10:29:47 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
2025-09-07T07:37:03.3252889Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-09-07T07:37:03.3253469Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-09-07T07:37:03.3262201Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:03.3262459Z env:
2025-09-07T07:37:03.3262611Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:03.3262782Z ##[endgroup]
2025-09-07T07:37:03.3334888Z ##[group]Run if systemctl is-active --quiet docker; then
2025-09-07T07:37:03.3335191Z if systemctl is-active --quiet docker; then
2025-09-07T07:37:03.3335438Z     echo "Docker daemon is running...";
2025-09-07T07:37:03.3335654Z else
2025-09-07T07:37:03.3335875Z     echo "Starting docker daemon..." && sudo systemctl start docker;
2025-09-07T07:37:03.3336139Z fi
2025-09-07T07:37:03.3343316Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:03.3343728Z env:
2025-09-07T07:37:03.3343871Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:03.3344044Z ##[endgroup]
2025-09-07T07:37:03.3429845Z Docker daemon is running...
2025-09-07T07:37:03.3462426Z ##[group]Run nick-fields/retry@v3.0.0
2025-09-07T07:37:03.3462646Z with:
2025-09-07T07:37:03.3462778Z   shell: bash
2025-09-07T07:37:03.3463123Z   timeout_minutes: 5
2025-09-07T07:37:03.3463289Z   max_attempts: 3
2025-09-07T07:37:03.3463454Z   retry_wait_seconds: 30
2025-09-07T07:37:03.3464826Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2025-09-07T07:37:03.3466148Z   polling_interval_seconds: 1
2025-09-07T07:37:03.3466333Z   warning_on_retry: true
2025-09-07T07:37:03.3466506Z   continue_on_error: false
2025-09-07T07:37:03.3466665Z env:
2025-09-07T07:37:03.3466809Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:03.3466977Z   AWS_RETRY_MODE: standard
2025-09-07T07:37:03.3467143Z   AWS_MAX_ATTEMPTS: 5
2025-09-07T07:37:03.3467304Z   AWS_DEFAULT_REGION: us-east-1
2025-09-07T07:37:03.3467481Z ##[endgroup]
2025-09-07T07:37:04.3632290Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-09-07T07:37:04.3632703Z Configure a credential helper to remove this warning. See
2025-09-07T07:37:04.3633075Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-09-07T07:37:04.3633314Z 
2025-09-07T07:37:04.3633398Z Login Succeeded
2025-09-07T07:37:04.5188871Z Command completed after 1 attempt(s).
2025-09-07T07:37:04.5243115Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-09-07T07:37:04.5243460Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-09-07T07:37:04.5243748Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-09-07T07:37:04.5253669Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:04.5253902Z env:
2025-09-07T07:37:04.5254052Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:04.5254225Z ##[endgroup]
2025-09-07T07:37:04.5347201Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-09-07T07:37:04.5347581Z # ignore expansion of "docker ps -q" since it could be empty
2025-09-07T07:37:04.5347853Z # shellcheck disable=SC2046
2025-09-07T07:37:04.5379437Z docker stop $(docker ps -q) || true
2025-09-07T07:37:04.5379757Z # Prune all of the docker images
2025-09-07T07:37:04.5379997Z docker system prune -af
2025-09-07T07:37:04.5387432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:04.5387662Z env:
2025-09-07T07:37:04.5387801Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:04.5387961Z ##[endgroup]
2025-09-07T07:37:04.5884831Z "docker stop" requires at least 1 argument.
2025-09-07T07:37:04.5885120Z See 'docker stop --help'.
2025-09-07T07:37:04.5885267Z 
2025-09-07T07:37:04.5885382Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-09-07T07:37:04.5885560Z 
2025-09-07T07:37:04.5885640Z Stop one or more running containers
2025-09-07T07:37:04.6243258Z Total reclaimed space: 0B
2025-09-07T07:37:04.6278734Z ##[group]Run set +e
2025-09-07T07:37:04.6278944Z set +e
2025-09-07T07:37:04.6279098Z set -x
2025-09-07T07:37:04.6279240Z 
2025-09-07T07:37:04.6279403Z PT_DOMAIN=download.pytorch.org
2025-09-07T07:37:04.6279758Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2025-09-07T07:37:04.6280397Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2025-09-07T07:37:04.6280715Z # one is returned at random
2025-09-07T07:37:04.6280966Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2025-09-07T07:37:04.6281201Z 
2025-09-07T07:37:04.6281497Z if [ -z "${RESOLVED_IP}" ]; then
2025-09-07T07:37:04.6281769Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2025-09-07T07:37:04.6282084Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2025-09-07T07:37:04.6282319Z 
2025-09-07T07:37:04.6282473Z   if [ -z "${RESOLVED_IP}" ]; then
2025-09-07T07:37:04.6282702Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2025-09-07T07:37:04.6282924Z     exit 1
2025-09-07T07:37:04.6283074Z   fi
2025-09-07T07:37:04.6283214Z fi
2025-09-07T07:37:04.6283340Z 
2025-09-07T07:37:04.6283507Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2025-09-07T07:37:04.6283735Z   # Clean up any old records first
2025-09-07T07:37:04.6283955Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2025-09-07T07:37:04.6284144Z fi
2025-09-07T07:37:04.6284276Z 
2025-09-07T07:37:04.6284473Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2025-09-07T07:37:04.6284713Z cat /etc/hosts
2025-09-07T07:37:04.6291944Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:04.6292181Z env:
2025-09-07T07:37:04.6292331Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:04.6292503Z ##[endgroup]
2025-09-07T07:37:04.6319974Z + PT_DOMAIN=download.pytorch.org
2025-09-07T07:37:04.6327331Z ++ dig -4 +short download.pytorch.org
2025-09-07T07:37:04.6328102Z ++ tail -n1
2025-09-07T07:37:04.6774999Z + RESOLVED_IP=18.160.10.28
2025-09-07T07:37:04.6776038Z + '[' -z 18.160.10.28 ']'
2025-09-07T07:37:04.6776320Z + grep -r download.pytorch.org /etc/hosts
2025-09-07T07:37:04.6796513Z + echo '18.160.10.28 download.pytorch.org'
2025-09-07T07:37:04.6797080Z + sudo tee -a /etc/hosts
2025-09-07T07:37:05.0761666Z 18.160.10.28 download.pytorch.org
2025-09-07T07:37:05.0778497Z + cat /etc/hosts
2025-09-07T07:37:05.0792016Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2025-09-07T07:37:05.0796731Z ::1         localhost6 localhost6.localdomain6
2025-09-07T07:37:05.0797009Z 18.160.10.28 download.pytorch.org
2025-09-07T07:37:05.0900730Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-09-07T07:37:05.0901039Z with:
2025-09-07T07:37:05.0901570Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.0902138Z   use-custom-docker-registry: true
2025-09-07T07:37:05.0902335Z   docker-build-dir: .ci/docker
2025-09-07T07:37:05.0902524Z   docker-build-script: ./build.sh
2025-09-07T07:37:05.0902731Z   working-directory: .
2025-09-07T07:37:05.0902973Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:05.0903435Z   force-push: false
2025-09-07T07:37:05.0903589Z env:
2025-09-07T07:37:05.0903732Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:05.0903901Z ##[endgroup]
2025-09-07T07:37:05.0917025Z ##[group]Run set -ex
2025-09-07T07:37:05.0917225Z set -ex
2025-09-07T07:37:05.0917369Z 
2025-09-07T07:37:05.0917642Z # If the docker build directory or the build script doesn't exist, the action will
2025-09-07T07:37:05.0918039Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-09-07T07:37:05.0918375Z # job could then download the pre-built image as usual
2025-09-07T07:37:05.0918794Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then
2025-09-07T07:37:05.0919327Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0919540Z else
2025-09-07T07:37:05.0919710Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0919982Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0920227Z 
2025-09-07T07:37:05.0920571Z   echo "Not using custom ECR registry.  Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..."
2025-09-07T07:37:05.0920951Z   exit 0
2025-09-07T07:37:05.0921092Z fi
2025-09-07T07:37:05.0921230Z 
2025-09-07T07:37:05.0921450Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-09-07T07:37:05.0921813Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-09-07T07:37:05.0922149Z   # use it as it is, but first let's extract the tag
2025-09-07T07:37:05.0922441Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-09-07T07:37:05.0922754Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0923046Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0923293Z else
2025-09-07T07:37:05.0923469Z   if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then
2025-09-07T07:37:05.0923699Z     CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:}
2025-09-07T07:37:05.0923943Z     DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*}
2025-09-07T07:37:05.0924148Z   fi
2025-09-07T07:37:05.0924427Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-09-07T07:37:05.0924784Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0925160Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0925572Z   echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.0925830Z fi
2025-09-07T07:37:05.0935257Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:05.0935488Z env:
2025-09-07T07:37:05.0935632Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:05.0935808Z   REPO_NAME: pytorch
2025-09-07T07:37:05.0936491Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.0937053Z   DOCKER_BUILD_DIR: .ci/docker
2025-09-07T07:37:05.0937245Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-09-07T07:37:05.0937489Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:05.0937817Z   USE_CUSTOM_DOCKER_REGISTRY: true
2025-09-07T07:37:05.0938000Z   CUSTOM_TAG_PREFIX: 
2025-09-07T07:37:05.0938156Z ##[endgroup]
2025-09-07T07:37:05.0966760Z + [[ -d .ci/docker ]]
2025-09-07T07:37:05.0966958Z + [[ -f .ci/docker/./build.sh ]]
2025-09-07T07:37:05.0967156Z + [[ true == \t\r\u\e ]]
2025-09-07T07:37:05.0967316Z + echo skip=false
2025-09-07T07:37:05.0968013Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-09-07T07:37:05.0974812Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.0975937Z ++ awk -F '[:,]' '{print $2}'
2025-09-07T07:37:05.1002695Z + DOCKER_TAG=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.1003473Z + echo docker-tag=pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.1004246Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.1027506Z ##[group]Run set +e
2025-09-07T07:37:05.1027721Z set +e
2025-09-07T07:37:05.1027873Z set -x
2025-09-07T07:37:05.1028017Z 
2025-09-07T07:37:05.1028150Z login() {
2025-09-07T07:37:05.1028451Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-09-07T07:37:05.1028777Z }
2025-09-07T07:37:05.1028915Z 
2025-09-07T07:37:05.1029046Z retry () {
2025-09-07T07:37:05.1029225Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-09-07T07:37:05.1029433Z }
2025-09-07T07:37:05.1029564Z 
2025-09-07T07:37:05.1029707Z retry login "${DOCKER_REGISTRY}"
2025-09-07T07:37:05.1029899Z 
2025-09-07T07:37:05.1030042Z START_TIME=$(date +%s)
2025-09-07T07:37:05.1030233Z # Wait up to 120 minutes
2025-09-07T07:37:05.1030479Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-09-07T07:37:05.1030770Z   # Check if image already exists, if it does then skip building it
2025-09-07T07:37:05.1031068Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-09-07T07:37:05.1031292Z     exit 0
2025-09-07T07:37:05.1031443Z   fi
2025-09-07T07:37:05.1031577Z 
2025-09-07T07:37:05.1031824Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-09-07T07:37:05.1032212Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-09-07T07:37:05.1032600Z   # latter, it will wait for the Docker images to become available before continuing
2025-09-07T07:37:05.1032914Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-09-07T07:37:05.1033159Z     # It's a Docker build job, let's build the image
2025-09-07T07:37:05.1033380Z     break
2025-09-07T07:37:05.1033533Z   else
2025-09-07T07:37:05.1033750Z     # It's a regular build job, wait for the image to become available
2025-09-07T07:37:05.1033995Z     sleep 300
2025-09-07T07:37:05.1034153Z   fi
2025-09-07T07:37:05.1034294Z done
2025-09-07T07:37:05.1034430Z 
2025-09-07T07:37:05.1034642Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-09-07T07:37:05.1035154Z # be empty.  The default action would be to continue rebuild the image
2025-09-07T07:37:05.1035474Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-09-07T07:37:05.1035752Z   # if we're on the base branch then use the parent commit
2025-09-07T07:37:05.1035999Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-09-07T07:37:05.1036189Z else
2025-09-07T07:37:05.1036395Z   # otherwise we're on a PR, so use the most recent base commit
2025-09-07T07:37:05.1036685Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-09-07T07:37:05.1036903Z fi
2025-09-07T07:37:05.1037034Z 
2025-09-07T07:37:05.1037183Z if [[ -z "${MERGE_BASE}" ]]; then
2025-09-07T07:37:05.1037405Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.1037613Z 
2025-09-07T07:37:05.1037890Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-09-07T07:37:05.1038215Z   exit 0
2025-09-07T07:37:05.1038360Z fi
2025-09-07T07:37:05.1038495Z 
2025-09-07T07:37:05.1038685Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-09-07T07:37:05.1039077Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-09-07T07:37:05.1039417Z   exit 1
2025-09-07T07:37:05.1039558Z fi
2025-09-07T07:37:05.1041151Z 
2025-09-07T07:37:05.1041375Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-09-07T07:37:05.1041763Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-09-07T07:37:05.1042115Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-09-07T07:37:05.1042524Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-09-07T07:37:05.1042974Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-09-07T07:37:05.1043241Z fi
2025-09-07T07:37:05.1043383Z 
2025-09-07T07:37:05.1043555Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-09-07T07:37:05.1050779Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:05.1051015Z env:
2025-09-07T07:37:05.1051161Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:05.1051342Z   DOCKER_BUILD_DIR: .ci/docker
2025-09-07T07:37:05.1051566Z   BASE_REVISION: 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:37:05.1052158Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.1052873Z   DOCKER_TAG: pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.1053321Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:05.1053557Z   DOCKER_PUSH: 
2025-09-07T07:37:05.1053710Z ##[endgroup]
2025-09-07T07:37:05.1077321Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:05.1077603Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:05.1080572Z + aws ecr get-login-password --region us-east-1
2025-09-07T07:37:05.1081838Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:05.5066371Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-09-07T07:37:05.5066802Z Configure a credential helper to remove this warning. See
2025-09-07T07:37:05.5067167Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-09-07T07:37:05.5067393Z 
2025-09-07T07:37:05.5067472Z Login Succeeded
2025-09-07T07:37:05.5088481Z ++ date +%s
2025-09-07T07:37:05.5100651Z + START_TIME=1757230625
2025-09-07T07:37:05.5105550Z ++ date +%s
2025-09-07T07:37:05.5117855Z + [[ 1757223425 -lt 1757230625 ]]
2025-09-07T07:37:05.5118449Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:05.7486365Z {
2025-09-07T07:37:05.7486615Z 	"schemaVersion": 2,
2025-09-07T07:37:05.7486923Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-09-07T07:37:05.7487212Z 	"config": {
2025-09-07T07:37:05.7487435Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-09-07T07:37:05.7487720Z 		"size": 30269,
2025-09-07T07:37:05.7487989Z 		"digest": "sha256:662d8c9dfc7db2f5d004293de4f2b7647941dee4c916479ef082d17fcdfd9c47"
2025-09-07T07:37:05.7488282Z 	},
2025-09-07T07:37:05.7488411Z 	"layers": [
2025-09-07T07:37:05.7488546Z 		{
2025-09-07T07:37:05.7488754Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7489009Z 			"size": 30448359,
2025-09-07T07:37:05.7489291Z 			"digest": "sha256:e6fdc8487bfe6d764301ef3634bc6c043841dc3ab05ca14f81e69c0f92562d46"
2025-09-07T07:37:05.7489576Z 		},
2025-09-07T07:37:05.7489696Z 		{
2025-09-07T07:37:05.7489894Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7490145Z 			"size": 1554,
2025-09-07T07:37:05.7490402Z 			"digest": "sha256:18a5ee5b0e2e283bf6d7b9c4c312b0448c75eff1c43446c22c5139a3aeec97fe"
2025-09-07T07:37:05.7490671Z 		},
2025-09-07T07:37:05.7490794Z 		{
2025-09-07T07:37:05.7491030Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7491672Z 			"size": 313297813,
2025-09-07T07:37:05.7491943Z 			"digest": "sha256:572424b92528ee46c84fdf3e9e1f5fd75e302621ad75dcf4257ad06778885094"
2025-09-07T07:37:05.7492235Z 		},
2025-09-07T07:37:05.7492360Z 		{
2025-09-07T07:37:05.7492565Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7492814Z 			"size": 793,
2025-09-07T07:37:05.7493076Z 			"digest": "sha256:1c35b7d4b67c6769f59f96a643d69c214c5b00291a4968cdd395eedbce82b9c0"
2025-09-07T07:37:05.7493351Z 		},
2025-09-07T07:37:05.7493470Z 		{
2025-09-07T07:37:05.7493669Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7493914Z 			"size": 106,
2025-09-07T07:37:05.7494164Z 			"digest": "sha256:68c20f3c23bb0bddb9b69e6ce2e45bcd5b1fcfd9b37dbe3de26b8a5f0e81ff13"
2025-09-07T07:37:05.7494450Z 		},
2025-09-07T07:37:05.7494570Z 		{
2025-09-07T07:37:05.7494762Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7494997Z 			"size": 704,
2025-09-07T07:37:05.7495237Z 			"digest": "sha256:7efa39950d3273a15b20bc5f6659373b2b4eb62e36328d96b289834c48d2e408"
2025-09-07T07:37:05.7495506Z 		},
2025-09-07T07:37:05.7495626Z 		{
2025-09-07T07:37:05.7495811Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7496050Z 			"size": 1214,
2025-09-07T07:37:05.7496303Z 			"digest": "sha256:a10eb16a7271e996ea9f1d769ba6bd2ec69358f2a79cf26649595a8cea38275f"
2025-09-07T07:37:05.7496580Z 		},
2025-09-07T07:37:05.7496699Z 		{
2025-09-07T07:37:05.7496882Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7497119Z 			"size": 485,
2025-09-07T07:37:05.7497350Z 			"digest": "sha256:7d52cf57965449440c17f257fe4c522f9685019961eaa9853d7c820cfe39f5cc"
2025-09-07T07:37:05.7497667Z 		},
2025-09-07T07:37:05.7497779Z 		{
2025-09-07T07:37:05.7497971Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7498219Z 			"size": 110343705,
2025-09-07T07:37:05.7498479Z 			"digest": "sha256:cb6a20fcf4e24ec2e1f72ecf361b26e058f3e6194947a9b3a25312223d43516e"
2025-09-07T07:37:05.7498752Z 		},
2025-09-07T07:37:05.7498871Z 		{
2025-09-07T07:37:05.7499060Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7499298Z 			"size": 4787,
2025-09-07T07:37:05.7499534Z 			"digest": "sha256:46fb6a8b3e1d4eac9b3a21577824410003ed38f194b4b1486b747e324b32ef6a"
2025-09-07T07:37:05.7499808Z 		},
2025-09-07T07:37:05.7500072Z 		{
2025-09-07T07:37:05.7500263Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7500492Z 			"size": 1709,
2025-09-07T07:37:05.7500741Z 			"digest": "sha256:5ad6977cc38e4ea8a6545d6a4fc0e2fdde705a7af96eb496cfe20f264fbc1e74"
2025-09-07T07:37:05.7501020Z 		},
2025-09-07T07:37:05.7501138Z 		{
2025-09-07T07:37:05.7501323Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7501563Z 			"size": 724,
2025-09-07T07:37:05.7501813Z 			"digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031"
2025-09-07T07:37:05.7502082Z 		},
2025-09-07T07:37:05.7502198Z 		{
2025-09-07T07:37:05.7502395Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7502642Z 			"size": 543,
2025-09-07T07:37:05.7502883Z 			"digest": "sha256:78243fdb9906cb588921ddaa67a3ca915aa9447ca675faac1a9ebc420a561d83"
2025-09-07T07:37:05.7503339Z 		},
2025-09-07T07:37:05.7503459Z 		{
2025-09-07T07:37:05.7503652Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7503895Z 			"size": 3395447162,
2025-09-07T07:37:05.7504143Z 			"digest": "sha256:6f70d5d50abaab8988f460b5590d92b6d1d340575ddee981662c24034d7d20af"
2025-09-07T07:37:05.7504415Z 		},
2025-09-07T07:37:05.7504536Z 		{
2025-09-07T07:37:05.7504725Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7504955Z 			"size": 32,
2025-09-07T07:37:05.7505197Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7505591Z 		},
2025-09-07T07:37:05.7505715Z 		{
2025-09-07T07:37:05.7505905Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7506150Z 			"size": 380,
2025-09-07T07:37:05.7506402Z 			"digest": "sha256:69715d3ad3c493436abde51f5a575e79f7d55b46c653f5607f3c7722ad9a05db"
2025-09-07T07:37:05.7506682Z 		},
2025-09-07T07:37:05.7506798Z 		{
2025-09-07T07:37:05.7506995Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7507240Z 			"size": 235844,
2025-09-07T07:37:05.7507494Z 			"digest": "sha256:7ace90c063f3f3ce8f04b541afe935088868930e5c074824af2b2c327779a3b5"
2025-09-07T07:37:05.7507765Z 		},
2025-09-07T07:37:05.7507882Z 		{
2025-09-07T07:37:05.7508076Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7508316Z 			"size": 230,
2025-09-07T07:37:05.7508552Z 			"digest": "sha256:acbd5447dd1406dab8e46234f6a034a75ad9794f76c24f817b0ecf28b6a69c78"
2025-09-07T07:37:05.7508826Z 		},
2025-09-07T07:37:05.7508963Z 		{
2025-09-07T07:37:05.7509159Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7509540Z 			"size": 3396092,
2025-09-07T07:37:05.7509798Z 			"digest": "sha256:744523d9b7f5a3e7abfc646c2d5222e7379024242430b93cb4b8093574e69022"
2025-09-07T07:37:05.7510069Z 		},
2025-09-07T07:37:05.7510192Z 		{
2025-09-07T07:37:05.7510386Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7510635Z 			"size": 1477,
2025-09-07T07:37:05.7510887Z 			"digest": "sha256:5bd615a7b945084e11bcb40190f9d6e50367297237146df7b008fa8c668f29c8"
2025-09-07T07:37:05.7511162Z 		},
2025-09-07T07:37:05.7511283Z 		{
2025-09-07T07:37:05.7511469Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7511709Z 			"size": 482,
2025-09-07T07:37:05.7511964Z 			"digest": "sha256:f4986a00e3aecf1d56beaada7aba8c49fbb3683db3c99790ab0aa4caaa34f76f"
2025-09-07T07:37:05.7512249Z 		},
2025-09-07T07:37:05.7512362Z 		{
2025-09-07T07:37:05.7512550Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7512787Z 			"size": 196,
2025-09-07T07:37:05.7513023Z 			"digest": "sha256:21902f6e4f8cb76c82e755b8fc9f72e1912bf925ab345ab5b4cc2210f4887a64"
2025-09-07T07:37:05.7513286Z 		},
2025-09-07T07:37:05.7513402Z 		{
2025-09-07T07:37:05.7513591Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7513827Z 			"size": 608,
2025-09-07T07:37:05.7514176Z 			"digest": "sha256:d80602abf3ccf0c0b527848a403dfde36e1cf1db1416852385feda5c44bf4363"
2025-09-07T07:37:05.7514454Z 		},
2025-09-07T07:37:05.7514573Z 		{
2025-09-07T07:37:05.7514761Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7514992Z 			"size": 226,
2025-09-07T07:37:05.7515232Z 			"digest": "sha256:3c51bf0bc362d34a17911f73c5146cbd668c4d1cf1b944cbf40a604d71cd623a"
2025-09-07T07:37:05.7515502Z 		},
2025-09-07T07:37:05.7515617Z 		{
2025-09-07T07:37:05.7515801Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7516039Z 			"size": 828,
2025-09-07T07:37:05.7516271Z 			"digest": "sha256:119ab3bceafa6f2cab4b1f71161195139792990263ee8de82230c6284f0ae20a"
2025-09-07T07:37:05.7516540Z 		},
2025-09-07T07:37:05.7516652Z 		{
2025-09-07T07:37:05.7516837Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7517072Z 			"size": 32,
2025-09-07T07:37:05.7517315Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7517581Z 		},
2025-09-07T07:37:05.7517702Z 		{
2025-09-07T07:37:05.7517890Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7518134Z 			"size": 104,
2025-09-07T07:37:05.7518370Z 			"digest": "sha256:af8eadc9eaabdaf6c5e01031d63061605327153e07568ddd159966ecea75cd07"
2025-09-07T07:37:05.7518645Z 		},
2025-09-07T07:37:05.7518762Z 		{
2025-09-07T07:37:05.7518948Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7519259Z 			"size": 1495,
2025-09-07T07:37:05.7519508Z 			"digest": "sha256:e7769b0d7a8262f3cc32a9d96080de5318dac3d2617e10508a167e689016e40c"
2025-09-07T07:37:05.7519783Z 		},
2025-09-07T07:37:05.7532207Z 		{
2025-09-07T07:37:05.7532450Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7532729Z 			"size": 453908015,
2025-09-07T07:37:05.7533014Z 			"digest": "sha256:ba263639b0f4634277ef3b8903e3457ac27ce012f1bbeeeeb773191c2c3b222b"
2025-09-07T07:37:05.7533310Z 		},
2025-09-07T07:37:05.7533435Z 		{
2025-09-07T07:37:05.7533653Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7533906Z 			"size": 164,
2025-09-07T07:37:05.7534163Z 			"digest": "sha256:a5ab7a280382a797dd5ba6a6716f667a231540ad1e0e7c8ba48bb24d5ab80ef0"
2025-09-07T07:37:05.7534445Z 		},
2025-09-07T07:37:05.7534565Z 		{
2025-09-07T07:37:05.7534766Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7535008Z 			"size": 346,
2025-09-07T07:37:05.7535259Z 			"digest": "sha256:80b2232d952f55c3662cffd657ba30fe825f08dfcc5bbea13e2bc6de4482b7e4"
2025-09-07T07:37:05.7535532Z 		},
2025-09-07T07:37:05.7535653Z 		{
2025-09-07T07:37:05.7535836Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7536079Z 			"size": 32,
2025-09-07T07:37:05.7536320Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7536602Z 		},
2025-09-07T07:37:05.7536725Z 		{
2025-09-07T07:37:05.7536911Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7537148Z 			"size": 106,
2025-09-07T07:37:05.7537391Z 			"digest": "sha256:cc93cd65e90f0a9c50194579c93e96897f4e582b9777a1c4d7df7b913ddcdded"
2025-09-07T07:37:05.7537737Z 		},
2025-09-07T07:37:05.7537852Z 		{
2025-09-07T07:37:05.7538042Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7538286Z 			"size": 425,
2025-09-07T07:37:05.7538529Z 			"digest": "sha256:0eed4c15712bc470dac7df87e33b3570a1510344019dd9cc0e95b8beb1f98372"
2025-09-07T07:37:05.7538795Z 		},
2025-09-07T07:37:05.7538919Z 		{
2025-09-07T07:37:05.7539106Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7539353Z 			"size": 19309387,
2025-09-07T07:37:05.7539594Z 			"digest": "sha256:092516f71fe325518f9737f105bcd65c40cd35c3019098889757e2c84c03c8a8"
2025-09-07T07:37:05.7539856Z 		},
2025-09-07T07:37:05.7540087Z 		{
2025-09-07T07:37:05.7540279Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7540512Z 			"size": 108,
2025-09-07T07:37:05.7540751Z 			"digest": "sha256:8c0825014a6270f765ff514da8583d55874f3278bef76e5617e29115f91ee654"
2025-09-07T07:37:05.7541021Z 		},
2025-09-07T07:37:05.7541141Z 		{
2025-09-07T07:37:05.7541324Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7541559Z 			"size": 636,
2025-09-07T07:37:05.7541806Z 			"digest": "sha256:8e0d2f63da0a8ff07657d7e06cdbc1ad9d5db95614d640a9f7a9aa8c30c9986d"
2025-09-07T07:37:05.7542085Z 		},
2025-09-07T07:37:05.7542195Z 		{
2025-09-07T07:37:05.7542380Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7542618Z 			"size": 724,
2025-09-07T07:37:05.7542850Z 			"digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031"
2025-09-07T07:37:05.7543108Z 		},
2025-09-07T07:37:05.7543228Z 		{
2025-09-07T07:37:05.7543419Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7543658Z 			"size": 148,
2025-09-07T07:37:05.7543886Z 			"digest": "sha256:73aae7958ba1a16c5f5625d39b06208e1def8c7816bb75028bf0845f553a5068"
2025-09-07T07:37:05.7544154Z 		},
2025-09-07T07:37:05.7544273Z 		{
2025-09-07T07:37:05.7544457Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7544685Z 			"size": 136,
2025-09-07T07:37:05.7544985Z 			"digest": "sha256:ac6077ec9fa50fc0822d387d2ee35e1b6f1f56612402fe7195378180b25087bc"
2025-09-07T07:37:05.7545386Z 		},
2025-09-07T07:37:05.7545508Z 		{
2025-09-07T07:37:05.7545690Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7545973Z + exit 0
2025-09-07T07:37:05.7546099Z 			"size": 140,
2025-09-07T07:37:05.7546341Z 			"digest": "sha256:bf4ee4e45e92ef179f7fc64e2c7c6755905a969c37cf82c39aafbadd9290ff04"
2025-09-07T07:37:05.7546631Z 		},
2025-09-07T07:37:05.7546749Z 		{
2025-09-07T07:37:05.7546940Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7547183Z 			"size": 18617175577,
2025-09-07T07:37:05.7547443Z 			"digest": "sha256:c1b766f9b961bcc863d6f89d623815fd7dfe9797ddcfd5d15ef06ffe7d177359"
2025-09-07T07:37:05.7547717Z 		},
2025-09-07T07:37:05.7547833Z 		{
2025-09-07T07:37:05.7548023Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7548263Z 			"size": 223,
2025-09-07T07:37:05.7548516Z 			"digest": "sha256:6e726ef07b5d5cfe2fb9f06d43fc931fc64c381fd37eaf0c169e0dd84796f152"
2025-09-07T07:37:05.7548799Z 		},
2025-09-07T07:37:05.7548918Z 		{
2025-09-07T07:37:05.7549109Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7549350Z 			"size": 274477524,
2025-09-07T07:37:05.7549593Z 			"digest": "sha256:364070434a64fa913f3907ada910a4051707e693e0e6124f57bc97aa57791da1"
2025-09-07T07:37:05.7549863Z 		},
2025-09-07T07:37:05.7549981Z 		{
2025-09-07T07:37:05.7550173Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7550410Z 			"size": 6451569004,
2025-09-07T07:37:05.7550665Z 			"digest": "sha256:71f708151a84685fc366b85e914dac9f5279313eff07358d79ecaaeecb0f1c42"
2025-09-07T07:37:05.7550929Z 		},
2025-09-07T07:37:05.7551049Z 		{
2025-09-07T07:37:05.7551231Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7551472Z 			"size": 129,
2025-09-07T07:37:05.7551715Z 			"digest": "sha256:622d8cfb39ea4dda608d2819c6a9de45df81b6f8319ee8ab4a24c36d81b9a132"
2025-09-07T07:37:05.7551995Z 		},
2025-09-07T07:37:05.7552104Z 		{
2025-09-07T07:37:05.7552291Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7552528Z 			"size": 778,
2025-09-07T07:37:05.7552762Z 			"digest": "sha256:284119a92cb13dacff06926444aab4f99756039acb48abba7b75d35c367ed3f1"
2025-09-07T07:37:05.7553025Z 		},
2025-09-07T07:37:05.7553147Z 		{
2025-09-07T07:37:05.7553334Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7553658Z 			"size": 724,
2025-09-07T07:37:05.7553886Z 			"digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031"
2025-09-07T07:37:05.7554155Z 		},
2025-09-07T07:37:05.7554273Z 		{
2025-09-07T07:37:05.7554465Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7554697Z 			"size": 140,
2025-09-07T07:37:05.7554928Z 			"digest": "sha256:96695940d842555623cfe4fb7b52e949423e8c8f383e55d02363e7e5c5804afa"
2025-09-07T07:37:05.7555193Z 		},
2025-09-07T07:37:05.7555317Z 		{
2025-09-07T07:37:05.7555496Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7555735Z 			"size": 32,
2025-09-07T07:37:05.7555981Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7556266Z 		},
2025-09-07T07:37:05.7556379Z 		{
2025-09-07T07:37:05.7556570Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7556812Z 			"size": 160,
2025-09-07T07:37:05.7557056Z 			"digest": "sha256:7ddca6c4c050460204097ba875dc0fa03eca6265122a18c0b8dc5504152aea53"
2025-09-07T07:37:05.7557323Z 		},
2025-09-07T07:37:05.7557440Z 		{
2025-09-07T07:37:05.7557626Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7557863Z 			"size": 1012,
2025-09-07T07:37:05.7558122Z 			"digest": "sha256:a95e1f2f1aadef03514a7cdbdac1fe83d4eebedbb80df9be868a223f27e1c263"
2025-09-07T07:37:05.7558422Z 		},
2025-09-07T07:37:05.7558535Z 		{
2025-09-07T07:37:05.7560266Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7560503Z 			"size": 724,
2025-09-07T07:37:05.7560735Z 			"digest": "sha256:da63046995a2e510b7146776371a14bff4b31002cc3ef0322e45a3932fba2031"
2025-09-07T07:37:05.7561000Z 		},
2025-09-07T07:37:05.7561118Z 		{
2025-09-07T07:37:05.7561300Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7561536Z 			"size": 135,
2025-09-07T07:37:05.7561773Z 			"digest": "sha256:8085756b0cc0f9588f23a73c27840a5dff48cc18c3a2f0311e4d1ef291855679"
2025-09-07T07:37:05.7562043Z 		},
2025-09-07T07:37:05.7562154Z 		{
2025-09-07T07:37:05.7562340Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7562577Z 			"size": 32,
2025-09-07T07:37:05.7562820Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7563087Z 		},
2025-09-07T07:37:05.7563204Z 		{
2025-09-07T07:37:05.7563389Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7563627Z 			"size": 158,
2025-09-07T07:37:05.7563856Z 			"digest": "sha256:7e9ff0c6f103b18756f01c60b4d57a951660f17bffb1810b330e3ff703caf216"
2025-09-07T07:37:05.7564130Z 		},
2025-09-07T07:37:05.7564247Z 		{
2025-09-07T07:37:05.7564434Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7564668Z 			"size": 1369,
2025-09-07T07:37:05.7564911Z 			"digest": "sha256:a625cbbc05b983aeb4c28702a4a5b65c68191ab1b8d17978f7d98cc17ddf3c52"
2025-09-07T07:37:05.7565187Z 		},
2025-09-07T07:37:05.7565307Z 		{
2025-09-07T07:37:05.7565487Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7565724Z 			"size": 32,
2025-09-07T07:37:05.7565958Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7566230Z 		},
2025-09-07T07:37:05.7566345Z 		{
2025-09-07T07:37:05.7566531Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7566769Z 			"size": 136,
2025-09-07T07:37:05.7567000Z 			"digest": "sha256:4e28486424310870c8d6815524440f17c6e0afe7572eaa173a811b98b4920bed"
2025-09-07T07:37:05.7567265Z 		},
2025-09-07T07:37:05.7567384Z 		{
2025-09-07T07:37:05.7567578Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7567817Z 			"size": 380,
2025-09-07T07:37:05.7568050Z 			"digest": "sha256:5e944f1ed1bef9442f5b1b86225d3958ea8f2f7f4c6aa7b92dc5d0c810c260bc"
2025-09-07T07:37:05.7568324Z 		},
2025-09-07T07:37:05.7568523Z 		{
2025-09-07T07:37:05.7568717Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7568947Z 			"size": 32,
2025-09-07T07:37:05.7569194Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7569470Z 		},
2025-09-07T07:37:05.7569589Z 		{
2025-09-07T07:37:05.7569770Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7570004Z 			"size": 104,
2025-09-07T07:37:05.7570244Z 			"digest": "sha256:41619248f604c60e038a02bfd462af96ee2996b77be5f59f05e9ac5fe4790e5a"
2025-09-07T07:37:05.7570519Z 		},
2025-09-07T07:37:05.7570635Z 		{
2025-09-07T07:37:05.7570822Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7571051Z 			"size": 407,
2025-09-07T07:37:05.7571293Z 			"digest": "sha256:be86f8c4f654b9ae64a20eb7f960e6ce4baa5b46e0a1f5e1312b11492a40bcd4"
2025-09-07T07:37:05.7571571Z 		},
2025-09-07T07:37:05.7571686Z 		{
2025-09-07T07:37:05.7571867Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7572095Z 			"size": 32,
2025-09-07T07:37:05.7572320Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7572592Z 		},
2025-09-07T07:37:05.7572712Z 		{
2025-09-07T07:37:05.7572897Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7573130Z 			"size": 109,
2025-09-07T07:37:05.7573364Z 			"digest": "sha256:ef1340e22a4bc8cf42e1d40961cb32d183cd3da8f0b785b5425c32ee067690c1"
2025-09-07T07:37:05.7573722Z 		},
2025-09-07T07:37:05.7573842Z 		{
2025-09-07T07:37:05.7574042Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7574288Z 			"size": 1897,
2025-09-07T07:37:05.7574546Z 			"digest": "sha256:da8d8b696333cbf6b9f339ab859639c905d6752d7e65fea14c23c3c2dcba553e"
2025-09-07T07:37:05.7574819Z 		},
2025-09-07T07:37:05.7574939Z 		{
2025-09-07T07:37:05.7575134Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7575376Z 			"size": 243443118,
2025-09-07T07:37:05.7575629Z 			"digest": "sha256:386b0c49c4982a821fb6f427fbc7d9c7d2012e97c96a514a9c7a09304e76b935"
2025-09-07T07:37:05.7575901Z 		},
2025-09-07T07:37:05.7576025Z 		{
2025-09-07T07:37:05.7576218Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7576454Z 			"size": 106,
2025-09-07T07:37:05.7576714Z 			"digest": "sha256:2b1d0ea7efe0bf86e86df804d2cddbf83b113fdecd03f3ddfca728da30546f34"
2025-09-07T07:37:05.7577004Z 		},
2025-09-07T07:37:05.7577133Z 		{
2025-09-07T07:37:05.7577314Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7577644Z 			"size": 163,
2025-09-07T07:37:05.7577890Z 			"digest": "sha256:04c04be7408f20625b1bd8454e5a08c91fcf04d4f79ab3ec1b75ae6b1824174d"
2025-09-07T07:37:05.7578199Z 		},
2025-09-07T07:37:05.7578308Z 		{
2025-09-07T07:37:05.7578510Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7578793Z 			"size": 7943,
2025-09-07T07:37:05.7579035Z 			"digest": "sha256:f8690caa3ac5e845f2dcc25ad12815b5c7452285c3838a87c780bd03ecf072a3"
2025-09-07T07:37:05.7579295Z 		},
2025-09-07T07:37:05.7579403Z 		{
2025-09-07T07:37:05.7579581Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7579807Z 			"size": 8074,
2025-09-07T07:37:05.7580033Z 			"digest": "sha256:2908d6baaa6b21331dee5f210472cae0874d22b98b0a35420cad4fd753ed215f"
2025-09-07T07:37:05.7580294Z 		},
2025-09-07T07:37:05.7580408Z 		{
2025-09-07T07:37:05.7580586Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7580817Z 			"size": 303,
2025-09-07T07:37:05.7581037Z 			"digest": "sha256:37e2336101eba2c73995d34431e4fae8782d9e9700c42621777922490b2158ed"
2025-09-07T07:37:05.7581289Z 		},
2025-09-07T07:37:05.7581398Z 		{
2025-09-07T07:37:05.7581573Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7581801Z 			"size": 32,
2025-09-07T07:37:05.7582139Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7582415Z 		},
2025-09-07T07:37:05.7582541Z 		{
2025-09-07T07:37:05.7582754Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7583006Z 			"size": 108,
2025-09-07T07:37:05.7583232Z 			"digest": "sha256:f1ac881fde33994861be4324231269058643168b9aee60c699552d0d92d965da"
2025-09-07T07:37:05.7583483Z 		},
2025-09-07T07:37:05.7583591Z 		{
2025-09-07T07:37:05.7583772Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7584002Z 			"size": 54145699,
2025-09-07T07:37:05.7584239Z 			"digest": "sha256:43b14c67347e2813c5f63e928c14db60dbb35c330ccc865510cf79739d8b78a1"
2025-09-07T07:37:05.7584496Z 		},
2025-09-07T07:37:05.7584604Z 		{
2025-09-07T07:37:05.7584781Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-09-07T07:37:05.7585005Z 			"size": 32,
2025-09-07T07:37:05.7585237Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-09-07T07:37:05.7585497Z 		}
2025-09-07T07:37:05.7585603Z 	]
2025-09-07T07:37:05.7585712Z }
2025-09-07T07:37:05.7606630Z ##[group]Run set -eux
2025-09-07T07:37:05.7606822Z set -eux
2025-09-07T07:37:05.7607080Z # It's ok if this steps fails, it would then be an anonymous user like what we used to have
2025-09-07T07:37:05.7607765Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin || true
2025-09-07T07:37:05.7616923Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:05.7617153Z env:
2025-09-07T07:37:05.7617298Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:05.7617468Z ##[endgroup]
2025-09-07T07:37:05.7646703Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-09-07T07:37:05.7648376Z + jq --raw-output .SecretString
2025-09-07T07:37:05.7649047Z + jq -r .docker_hub_readonly_token
2025-09-07T07:37:05.7650461Z + docker login --username pytorchbot --password-stdin
2025-09-07T07:37:06.2192682Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-09-07T07:37:06.2193109Z Configure a credential helper to remove this warning. See
2025-09-07T07:37:06.2193488Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-09-07T07:37:06.2193735Z 
2025-09-07T07:37:06.2193862Z Login Succeeded
2025-09-07T07:37:06.2277839Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:}
2025-09-07T07:37:06.2278103Z tag=${ECR_DOCKER_IMAGE##*:}
2025-09-07T07:37:06.2278362Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2025-09-07T07:37:06.2286620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:06.2286856Z env:
2025-09-07T07:37:06.2287011Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:06.2287568Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.2288104Z ##[endgroup]
2025-09-07T07:37:06.2316580Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.2352399Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-09-07T07:37:06.2352694Z with:
2025-09-07T07:37:06.2353256Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.2353926Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:06.2354181Z env:
2025-09-07T07:37:06.2354338Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:06.2354518Z ##[endgroup]
2025-09-07T07:37:06.2365149Z ##[group]Run set -x
2025-09-07T07:37:06.2365342Z set -x
2025-09-07T07:37:06.2365485Z set +e
2025-09-07T07:37:06.2365625Z 
2025-09-07T07:37:06.2365754Z login() {
2025-09-07T07:37:06.2366056Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-09-07T07:37:06.2366364Z }
2025-09-07T07:37:06.2366501Z 
2025-09-07T07:37:06.2366873Z retry () {
2025-09-07T07:37:06.2367054Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-09-07T07:37:06.2367248Z }
2025-09-07T07:37:06.2367383Z 
2025-09-07T07:37:06.2367543Z retry login "${DOCKER_REGISTRY}"
2025-09-07T07:37:06.2367734Z 
2025-09-07T07:37:06.2368029Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-09-07T07:37:06.2368423Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-09-07T07:37:06.2368653Z 
2025-09-07T07:37:06.2368789Z set -e
2025-09-07T07:37:06.2369000Z # ignore output since only exit code is used for conditional
2025-09-07T07:37:06.2369302Z # only pull docker image if it's not available locally
2025-09-07T07:37:06.2369629Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-09-07T07:37:06.2369936Z   retry docker pull "${DOCKER_IMAGE}"
2025-09-07T07:37:06.2370133Z fi
2025-09-07T07:37:06.2377149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:37:06.2377386Z env:
2025-09-07T07:37:06.2377743Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:37:06.2378282Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.2378876Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:06.2379111Z ##[endgroup]
2025-09-07T07:37:06.2405048Z + set +e
2025-09-07T07:37:06.2405273Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:06.2405559Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:06.2409291Z + aws ecr get-login-password --region us-east-1
2025-09-07T07:37:06.2410589Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-09-07T07:37:06.6548096Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-09-07T07:37:06.6548489Z Configure a credential helper to remove this warning. See
2025-09-07T07:37:06.6548846Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-09-07T07:37:06.6549097Z 
2025-09-07T07:37:06.6550276Z Login Succeeded
2025-09-07T07:37:06.6576512Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.6577159Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-09-07T07:37:06.8644129Z + IMAGE_SIZE=28579.020259857178
2025-09-07T07:37:06.8644437Z + echo 'Compressed size of image in MB: 28579.020259857178'
2025-09-07T07:37:06.8644675Z + set -e
2025-09-07T07:37:06.8645542Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.8646163Z Compressed size of image in MB: 28579.020259857178
2025-09-07T07:37:06.8804330Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:06.8805240Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:37:07.0876065Z pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77: Pulling from pytorch/ci-image
2025-09-07T07:37:07.0876619Z e6fdc8487bfe: Pulling fs layer
2025-09-07T07:37:07.0876830Z 18a5ee5b0e2e: Pulling fs layer
2025-09-07T07:37:07.0877021Z 572424b92528: Pulling fs layer
2025-09-07T07:37:07.0877196Z 1c35b7d4b67c: Pulling fs layer
2025-09-07T07:37:07.0877360Z 68c20f3c23bb: Pulling fs layer
2025-09-07T07:37:07.0877530Z 7efa39950d32: Pulling fs layer
2025-09-07T07:37:07.0877731Z a10eb16a7271: Pulling fs layer
2025-09-07T07:37:07.0877903Z 7d52cf579654: Pulling fs layer
2025-09-07T07:37:07.0878066Z cb6a20fcf4e2: Pulling fs layer
2025-09-07T07:37:07.0878237Z 46fb6a8b3e1d: Pulling fs layer
2025-09-07T07:37:07.0878421Z 5ad6977cc38e: Pulling fs layer
2025-09-07T07:37:07.0878592Z da63046995a2: Pulling fs layer
2025-09-07T07:37:07.0878752Z 78243fdb9906: Pulling fs layer
2025-09-07T07:37:07.0878918Z 6f70d5d50aba: Pulling fs layer
2025-09-07T07:37:07.0879091Z 4f4fb700ef54: Pulling fs layer
2025-09-07T07:37:07.0879262Z 69715d3ad3c4: Pulling fs layer
2025-09-07T07:37:07.0879422Z 7ace90c063f3: Pulling fs layer
2025-09-07T07:37:07.0879586Z acbd5447dd14: Pulling fs layer
2025-09-07T07:37:07.0879751Z 744523d9b7f5: Pulling fs layer
2025-09-07T07:37:07.0879914Z 5bd615a7b945: Pulling fs layer
2025-09-07T07:37:07.0880077Z f4986a00e3ae: Pulling fs layer
2025-09-07T07:37:07.0880255Z 21902f6e4f8c: Pulling fs layer
2025-09-07T07:37:07.0880422Z d80602abf3cc: Pulling fs layer
2025-09-07T07:37:07.0880591Z 3c51bf0bc362: Pulling fs layer
2025-09-07T07:37:07.0880778Z 119ab3bceafa: Pulling fs layer
2025-09-07T07:37:07.0880949Z af8eadc9eaab: Pulling fs layer
2025-09-07T07:37:07.0881116Z e7769b0d7a82: Pulling fs layer
2025-09-07T07:37:07.0881530Z ba263639b0f4: Pulling fs layer
2025-09-07T07:37:07.0881686Z a5ab7a280382: Pulling fs layer
2025-09-07T07:37:07.0881853Z 80b2232d952f: Pulling fs layer
2025-09-07T07:37:07.0882019Z cc93cd65e90f: Pulling fs layer
2025-09-07T07:37:07.0882185Z 0eed4c15712b: Pulling fs layer
2025-09-07T07:37:07.0882363Z 092516f71fe3: Pulling fs layer
2025-09-07T07:37:07.0882533Z 8c0825014a62: Pulling fs layer
2025-09-07T07:37:07.0882706Z 8e0d2f63da0a: Pulling fs layer
2025-09-07T07:37:07.0882889Z 73aae7958ba1: Pulling fs layer
2025-09-07T07:37:07.0883064Z ac6077ec9fa5: Pulling fs layer
2025-09-07T07:37:07.0883238Z bf4ee4e45e92: Pulling fs layer
2025-09-07T07:37:07.0883414Z c1b766f9b961: Pulling fs layer
2025-09-07T07:37:07.0883581Z 6e726ef07b5d: Pulling fs layer
2025-09-07T07:37:07.0883747Z 364070434a64: Pulling fs layer
2025-09-07T07:37:07.0883908Z 71f708151a84: Pulling fs layer
2025-09-07T07:37:07.0884102Z 622d8cfb39ea: Pulling fs layer
2025-09-07T07:37:07.0884274Z 284119a92cb1: Pulling fs layer
2025-09-07T07:37:07.0884433Z 96695940d842: Pulling fs layer
2025-09-07T07:37:07.0884608Z 7ddca6c4c050: Pulling fs layer
2025-09-07T07:37:07.0884776Z a95e1f2f1aad: Pulling fs layer
2025-09-07T07:37:07.0884941Z 8085756b0cc0: Pulling fs layer
2025-09-07T07:37:07.0885101Z 7e9ff0c6f103: Pulling fs layer
2025-09-07T07:37:07.0885271Z a625cbbc05b9: Pulling fs layer
2025-09-07T07:37:07.0885438Z 4e2848642431: Pulling fs layer
2025-09-07T07:37:07.0885651Z 5e944f1ed1be: Pulling fs layer
2025-09-07T07:37:07.0885818Z 41619248f604: Pulling fs layer
2025-09-07T07:37:07.0885974Z be86f8c4f654: Pulling fs layer
2025-09-07T07:37:07.0886141Z ef1340e22a4b: Pulling fs layer
2025-09-07T07:37:07.0886305Z da8d8b696333: Pulling fs layer
2025-09-07T07:37:07.0886484Z 386b0c49c498: Pulling fs layer
2025-09-07T07:37:07.0888075Z 2b1d0ea7efe0: Pulling fs layer
2025-09-07T07:37:07.0888257Z 04c04be7408f: Pulling fs layer
2025-09-07T07:37:07.0888422Z f8690caa3ac5: Pulling fs layer
2025-09-07T07:37:07.0888582Z 2908d6baaa6b: Pulling fs layer
2025-09-07T07:37:07.0888769Z 37e2336101eb: Pulling fs layer
2025-09-07T07:37:07.0888941Z f1ac881fde33: Pulling fs layer
2025-09-07T07:37:07.0889111Z 43b14c67347e: Pulling fs layer
2025-09-07T07:37:07.0889268Z a95e1f2f1aad: Waiting
2025-09-07T07:37:07.0889429Z 8e0d2f63da0a: Waiting
2025-09-07T07:37:07.0889579Z a10eb16a7271: Waiting
2025-09-07T07:37:07.0889718Z 7d52cf579654: Waiting
2025-09-07T07:37:07.0889862Z acbd5447dd14: Waiting
2025-09-07T07:37:07.0890003Z 46fb6a8b3e1d: Waiting
2025-09-07T07:37:07.0890152Z 5ad6977cc38e: Waiting
2025-09-07T07:37:07.0900324Z 6f70d5d50aba: Waiting
2025-09-07T07:37:07.0900547Z 5bd615a7b945: Waiting
2025-09-07T07:37:07.0900707Z f4986a00e3ae: Waiting
2025-09-07T07:37:07.0900865Z 73aae7958ba1: Waiting
2025-09-07T07:37:07.0901020Z 21902f6e4f8c: Waiting
2025-09-07T07:37:07.0901177Z da63046995a2: Waiting
2025-09-07T07:37:07.0901320Z 0eed4c15712b: Waiting
2025-09-07T07:37:07.0901468Z 69715d3ad3c4: Waiting
2025-09-07T07:37:07.0901611Z 78243fdb9906: Waiting
2025-09-07T07:37:07.0901762Z 1c35b7d4b67c: Waiting
2025-09-07T07:37:07.0901900Z cb6a20fcf4e2: Waiting
2025-09-07T07:37:07.0902043Z 8c0825014a62: Waiting
2025-09-07T07:37:07.0902185Z ba263639b0f4: Waiting
2025-09-07T07:37:07.0902322Z bf4ee4e45e92: Waiting
2025-09-07T07:37:07.0902466Z a5ab7a280382: Waiting
2025-09-07T07:37:07.0902609Z ac6077ec9fa5: Waiting
2025-09-07T07:37:07.0902757Z d80602abf3cc: Waiting
2025-09-07T07:37:07.0902898Z 744523d9b7f5: Waiting
2025-09-07T07:37:07.0903220Z 4e2848642431: Waiting
2025-09-07T07:37:07.0903365Z e7769b0d7a82: Waiting
2025-09-07T07:37:07.0903512Z 7e9ff0c6f103: Waiting
2025-09-07T07:37:07.0903647Z 092516f71fe3: Waiting
2025-09-07T07:37:07.0903792Z cc93cd65e90f: Waiting
2025-09-07T07:37:07.0903938Z 7ace90c063f3: Waiting
2025-09-07T07:37:07.0904083Z 68c20f3c23bb: Waiting
2025-09-07T07:37:07.0904229Z 7ddca6c4c050: Waiting
2025-09-07T07:37:07.0904372Z 7efa39950d32: Waiting
2025-09-07T07:37:07.0904522Z 119ab3bceafa: Waiting
2025-09-07T07:37:07.0904669Z 4f4fb700ef54: Waiting
2025-09-07T07:37:07.0904993Z 5e944f1ed1be: Waiting
2025-09-07T07:37:07.0905144Z 80b2232d952f: Waiting
2025-09-07T07:37:07.0905298Z 8085756b0cc0: Waiting
2025-09-07T07:37:07.0905448Z 622d8cfb39ea: Waiting
2025-09-07T07:37:07.0905590Z a625cbbc05b9: Waiting
2025-09-07T07:37:07.0905733Z 284119a92cb1: Waiting
2025-09-07T07:37:07.0905872Z be86f8c4f654: Waiting
2025-09-07T07:37:07.0906018Z af8eadc9eaab: Waiting
2025-09-07T07:37:07.0906161Z 37e2336101eb: Waiting
2025-09-07T07:37:07.0906322Z f8690caa3ac5: Waiting
2025-09-07T07:37:07.0906472Z 386b0c49c498: Waiting
2025-09-07T07:37:07.0906616Z 41619248f604: Waiting
2025-09-07T07:37:07.0906754Z f1ac881fde33: Waiting
2025-09-07T07:37:07.0906899Z 96695940d842: Waiting
2025-09-07T07:37:07.0907045Z da8d8b696333: Waiting
2025-09-07T07:37:07.0907189Z 71f708151a84: Waiting
2025-09-07T07:37:07.0907342Z 2908d6baaa6b: Waiting
2025-09-07T07:37:07.0907503Z 43b14c67347e: Waiting
2025-09-07T07:37:07.0907653Z 6e726ef07b5d: Waiting
2025-09-07T07:37:07.0907794Z 04c04be7408f: Waiting
2025-09-07T07:37:07.0907940Z 364070434a64: Waiting
2025-09-07T07:37:07.1608606Z 18a5ee5b0e2e: Download complete
2025-09-07T07:37:07.2489477Z 1c35b7d4b67c: Verifying Checksum
2025-09-07T07:37:07.2489745Z 1c35b7d4b67c: Download complete
2025-09-07T07:37:07.3473779Z 68c20f3c23bb: Verifying Checksum
2025-09-07T07:37:07.3474010Z 68c20f3c23bb: Download complete
2025-09-07T07:37:07.4344906Z e6fdc8487bfe: Verifying Checksum
2025-09-07T07:37:07.4345166Z e6fdc8487bfe: Download complete
2025-09-07T07:37:07.4394786Z 7efa39950d32: Verifying Checksum
2025-09-07T07:37:07.4395041Z 7efa39950d32: Download complete
2025-09-07T07:37:07.5096948Z a10eb16a7271: Verifying Checksum
2025-09-07T07:37:07.5097230Z a10eb16a7271: Download complete
2025-09-07T07:37:07.5188741Z 7d52cf579654: Verifying Checksum
2025-09-07T07:37:07.5188943Z 7d52cf579654: Download complete
2025-09-07T07:37:07.6123062Z 46fb6a8b3e1d: Verifying Checksum
2025-09-07T07:37:07.6123343Z 46fb6a8b3e1d: Download complete
2025-09-07T07:37:07.6887444Z 5ad6977cc38e: Download complete
2025-09-07T07:37:07.7743006Z da63046995a2: Verifying Checksum
2025-09-07T07:37:07.7743293Z da63046995a2: Download complete
2025-09-07T07:37:07.8660683Z 78243fdb9906: Verifying Checksum
2025-09-07T07:37:07.8660925Z 78243fdb9906: Download complete
2025-09-07T07:37:08.1079904Z e6fdc8487bfe: Pull complete
2025-09-07T07:37:08.1183627Z 18a5ee5b0e2e: Pull complete
2025-09-07T07:37:08.6747773Z cb6a20fcf4e2: Verifying Checksum
2025-09-07T07:37:08.6748018Z cb6a20fcf4e2: Download complete
2025-09-07T07:37:08.6814298Z 4f4fb700ef54: Verifying Checksum
2025-09-07T07:37:08.6814479Z 4f4fb700ef54: Download complete
2025-09-07T07:37:08.7766645Z 69715d3ad3c4: Verifying Checksum
2025-09-07T07:37:08.7766886Z 69715d3ad3c4: Download complete
2025-09-07T07:37:08.8655224Z 7ace90c063f3: Verifying Checksum
2025-09-07T07:37:08.8655475Z 7ace90c063f3: Download complete
2025-09-07T07:37:08.9641292Z acbd5447dd14: Verifying Checksum
2025-09-07T07:37:08.9641512Z acbd5447dd14: Download complete
2025-09-07T07:37:09.0606754Z 744523d9b7f5: Verifying Checksum
2025-09-07T07:37:09.0607005Z 744523d9b7f5: Download complete
2025-09-07T07:37:09.1545073Z 5bd615a7b945: Verifying Checksum
2025-09-07T07:37:09.1545341Z 5bd615a7b945: Download complete
2025-09-07T07:37:09.2244894Z f4986a00e3ae: Verifying Checksum
2025-09-07T07:37:09.2245130Z f4986a00e3ae: Download complete
2025-09-07T07:37:09.3196233Z 21902f6e4f8c: Verifying Checksum
2025-09-07T07:37:09.3196455Z 21902f6e4f8c: Download complete
2025-09-07T07:37:09.3953002Z d80602abf3cc: Verifying Checksum
2025-09-07T07:37:09.3953252Z d80602abf3cc: Download complete
2025-09-07T07:37:09.4842869Z 3c51bf0bc362: Download complete
2025-09-07T07:37:09.5500360Z 119ab3bceafa: Verifying Checksum
2025-09-07T07:37:09.5500577Z 119ab3bceafa: Download complete
2025-09-07T07:37:09.6263797Z af8eadc9eaab: Verifying Checksum
2025-09-07T07:37:09.6264033Z af8eadc9eaab: Download complete
2025-09-07T07:37:09.7000500Z e7769b0d7a82: Verifying Checksum
2025-09-07T07:37:09.7000726Z e7769b0d7a82: Download complete
2025-09-07T07:37:10.2820055Z 572424b92528: Verifying Checksum
2025-09-07T07:37:10.2820548Z 572424b92528: Download complete
2025-09-07T07:37:10.3639749Z a5ab7a280382: Verifying Checksum
2025-09-07T07:37:10.3639982Z a5ab7a280382: Download complete
2025-09-07T07:37:10.4466715Z 80b2232d952f: Verifying Checksum
2025-09-07T07:37:10.4466971Z 80b2232d952f: Download complete
2025-09-07T07:37:10.5384570Z cc93cd65e90f: Download complete
2025-09-07T07:37:10.6188400Z 0eed4c15712b: Verifying Checksum
2025-09-07T07:37:10.6188674Z 0eed4c15712b: Download complete
2025-09-07T07:37:10.8749679Z 092516f71fe3: Verifying Checksum
2025-09-07T07:37:10.8749915Z 092516f71fe3: Download complete
2025-09-07T07:37:10.9639085Z 8c0825014a62: Verifying Checksum
2025-09-07T07:37:10.9639325Z 8c0825014a62: Download complete
2025-09-07T07:37:11.0563651Z 8e0d2f63da0a: Verifying Checksum
2025-09-07T07:37:11.0563941Z 8e0d2f63da0a: Download complete
2025-09-07T07:37:11.1403893Z 73aae7958ba1: Verifying Checksum
2025-09-07T07:37:11.1404156Z 73aae7958ba1: Download complete
2025-09-07T07:37:11.2000474Z ac6077ec9fa5: Verifying Checksum
2025-09-07T07:37:11.2000722Z ac6077ec9fa5: Download complete
2025-09-07T07:37:11.2952614Z bf4ee4e45e92: Verifying Checksum
2025-09-07T07:37:11.2952885Z bf4ee4e45e92: Download complete
2025-09-07T07:37:14.3021466Z ba263639b0f4: Verifying Checksum
2025-09-07T07:37:14.3021749Z ba263639b0f4: Download complete
2025-09-07T07:37:14.3944367Z 6e726ef07b5d: Verifying Checksum
2025-09-07T07:37:14.3944609Z 6e726ef07b5d: Download complete
2025-09-07T07:37:16.5148185Z 572424b92528: Pull complete
2025-09-07T07:37:16.9136786Z 1c35b7d4b67c: Pull complete
2025-09-07T07:37:17.2051075Z 364070434a64: Verifying Checksum
2025-09-07T07:37:17.2051305Z 364070434a64: Download complete
2025-09-07T07:37:17.2652079Z 68c20f3c23bb: Pull complete
2025-09-07T07:37:17.6810200Z 7efa39950d32: Pull complete
2025-09-07T07:37:18.2043413Z a10eb16a7271: Pull complete
2025-09-07T07:37:18.7471748Z 7d52cf579654: Pull complete
2025-09-07T07:37:20.8591432Z cb6a20fcf4e2: Pull complete
2025-09-07T07:37:21.2300174Z 46fb6a8b3e1d: Pull complete
2025-09-07T07:37:21.6408070Z 5ad6977cc38e: Pull complete
2025-09-07T07:37:22.0704605Z da63046995a2: Pull complete
2025-09-07T07:37:22.5506864Z 78243fdb9906: Pull complete
2025-09-07T07:37:41.8836535Z 6f70d5d50aba: Verifying Checksum
2025-09-07T07:37:41.8836814Z 6f70d5d50aba: Download complete
2025-09-07T07:37:41.9656302Z 622d8cfb39ea: Verifying Checksum
2025-09-07T07:37:41.9656596Z 622d8cfb39ea: Download complete
2025-09-07T07:37:42.0330911Z 284119a92cb1: Verifying Checksum
2025-09-07T07:37:42.0331133Z 284119a92cb1: Download complete
2025-09-07T07:37:42.1132229Z 96695940d842: Verifying Checksum
2025-09-07T07:37:42.1132451Z 96695940d842: Download complete
2025-09-07T07:37:42.1769849Z 7ddca6c4c050: Verifying Checksum
2025-09-07T07:37:42.1770062Z 7ddca6c4c050: Download complete
2025-09-07T07:37:42.3018219Z a95e1f2f1aad: Verifying Checksum
2025-09-07T07:37:42.3018447Z a95e1f2f1aad: Download complete
2025-09-07T07:37:42.3944465Z 8085756b0cc0: Verifying Checksum
2025-09-07T07:37:42.3944703Z 8085756b0cc0: Download complete
2025-09-07T07:37:42.4736745Z 7e9ff0c6f103: Verifying Checksum
2025-09-07T07:37:42.4736980Z 7e9ff0c6f103: Download complete
2025-09-07T07:37:42.5403879Z a625cbbc05b9: Verifying Checksum
2025-09-07T07:37:42.5404105Z a625cbbc05b9: Download complete
2025-09-07T07:37:42.6130498Z 4e2848642431: Verifying Checksum
2025-09-07T07:37:42.6130741Z 4e2848642431: Download complete
2025-09-07T07:37:42.6931818Z 5e944f1ed1be: Verifying Checksum
2025-09-07T07:37:42.6932098Z 5e944f1ed1be: Download complete
2025-09-07T07:37:42.7789939Z 41619248f604: Verifying Checksum
2025-09-07T07:37:42.7790163Z 41619248f604: Download complete
2025-09-07T07:37:42.8799479Z be86f8c4f654: Verifying Checksum
2025-09-07T07:37:42.8799720Z be86f8c4f654: Download complete
2025-09-07T07:37:42.9683349Z ef1340e22a4b: Verifying Checksum
2025-09-07T07:37:42.9683624Z ef1340e22a4b: Download complete
2025-09-07T07:37:43.0617048Z da8d8b696333: Download complete
2025-09-07T07:37:45.5395487Z 386b0c49c498: Verifying Checksum
2025-09-07T07:37:45.5396265Z 386b0c49c498: Download complete
2025-09-07T07:37:45.6249265Z 2b1d0ea7efe0: Verifying Checksum
2025-09-07T07:37:45.6249521Z 2b1d0ea7efe0: Download complete
2025-09-07T07:37:45.6927621Z 04c04be7408f: Download complete
2025-09-07T07:37:45.7666809Z f8690caa3ac5: Download complete
2025-09-07T07:37:45.8330740Z 2908d6baaa6b: Verifying Checksum
2025-09-07T07:37:45.8330965Z 2908d6baaa6b: Download complete
2025-09-07T07:37:45.9792002Z 37e2336101eb: Verifying Checksum
2025-09-07T07:37:45.9792263Z 37e2336101eb: Download complete
2025-09-07T07:37:46.0585158Z f1ac881fde33: Download complete
2025-09-07T07:37:46.6681779Z 43b14c67347e: Verifying Checksum
2025-09-07T07:37:46.6682034Z 43b14c67347e: Download complete
2025-09-07T07:38:18.2569295Z 6f70d5d50aba: Pull complete
2025-09-07T07:38:18.7654563Z 4f4fb700ef54: Pull complete
2025-09-07T07:38:19.2038954Z 69715d3ad3c4: Pull complete
2025-09-07T07:38:19.7085737Z 7ace90c063f3: Pull complete
2025-09-07T07:38:20.1512568Z acbd5447dd14: Pull complete
2025-09-07T07:38:20.5793927Z 744523d9b7f5: Pull complete
2025-09-07T07:38:21.0230132Z 5bd615a7b945: Pull complete
2025-09-07T07:38:21.3319535Z f4986a00e3ae: Pull complete
2025-09-07T07:38:21.4092806Z 21902f6e4f8c: Pull complete
2025-09-07T07:38:21.5116558Z d80602abf3cc: Pull complete
2025-09-07T07:38:21.7974137Z 3c51bf0bc362: Pull complete
2025-09-07T07:38:21.8785484Z 71f708151a84: Verifying Checksum
2025-09-07T07:38:21.8785731Z 71f708151a84: Download complete
2025-09-07T07:38:22.2262062Z 119ab3bceafa: Pull complete
2025-09-07T07:38:22.9059793Z af8eadc9eaab: Pull complete
2025-09-07T07:38:23.2412022Z e7769b0d7a82: Pull complete
2025-09-07T07:38:28.7108976Z ba263639b0f4: Pull complete
2025-09-07T07:38:28.9369406Z a5ab7a280382: Pull complete
2025-09-07T07:38:29.2910877Z 80b2232d952f: Pull complete
2025-09-07T07:38:30.2807257Z cc93cd65e90f: Pull complete
2025-09-07T07:38:30.7226098Z 0eed4c15712b: Pull complete
2025-09-07T07:38:31.3945911Z 092516f71fe3: Pull complete
2025-09-07T07:38:31.9311261Z 8c0825014a62: Pull complete
2025-09-07T07:38:32.4647867Z 8e0d2f63da0a: Pull complete
2025-09-07T07:38:33.5231979Z 73aae7958ba1: Pull complete
2025-09-07T07:38:34.0487012Z ac6077ec9fa5: Pull complete
2025-09-07T07:38:34.5842570Z bf4ee4e45e92: Pull complete
2025-09-07T07:40:17.5103539Z c1b766f9b961: Verifying Checksum
2025-09-07T07:40:17.5103790Z c1b766f9b961: Download complete
2025-09-07T07:42:05.4316781Z c1b766f9b961: Pull complete
2025-09-07T07:42:05.9032320Z 6e726ef07b5d: Pull complete
2025-09-07T07:42:07.1109072Z 364070434a64: Pull complete
2025-09-07T07:43:14.3805641Z 71f708151a84: Pull complete
2025-09-07T07:43:14.8246549Z 622d8cfb39ea: Pull complete
2025-09-07T07:43:15.2830246Z 284119a92cb1: Pull complete
2025-09-07T07:43:16.2299932Z 96695940d842: Pull complete
2025-09-07T07:43:17.2300658Z 7ddca6c4c050: Pull complete
2025-09-07T07:43:17.6875382Z a95e1f2f1aad: Pull complete
2025-09-07T07:43:18.6914859Z 8085756b0cc0: Pull complete
2025-09-07T07:43:19.7368229Z 7e9ff0c6f103: Pull complete
2025-09-07T07:43:20.1650045Z a625cbbc05b9: Pull complete
2025-09-07T07:43:20.9255587Z 4e2848642431: Pull complete
2025-09-07T07:43:21.2464870Z 5e944f1ed1be: Pull complete
2025-09-07T07:43:22.2106296Z 41619248f604: Pull complete
2025-09-07T07:43:22.6109513Z be86f8c4f654: Pull complete
2025-09-07T07:43:23.3287122Z ef1340e22a4b: Pull complete
2025-09-07T07:43:23.8259615Z da8d8b696333: Pull complete
2025-09-07T07:43:29.1804076Z 386b0c49c498: Pull complete
2025-09-07T07:43:29.7143003Z 2b1d0ea7efe0: Pull complete
2025-09-07T07:43:30.1170180Z 04c04be7408f: Pull complete
2025-09-07T07:43:30.5475300Z f8690caa3ac5: Pull complete
2025-09-07T07:43:31.0887855Z 2908d6baaa6b: Pull complete
2025-09-07T07:43:31.4882558Z 37e2336101eb: Pull complete
2025-09-07T07:43:32.4272942Z f1ac881fde33: Pull complete
2025-09-07T07:43:34.0058608Z 43b14c67347e: Pull complete
2025-09-07T07:43:34.4850591Z Digest: sha256:383efb45082f20b8c808cb0ba4df693a01359592233f641f1f486911ac320a9a
2025-09-07T07:43:34.5259635Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:43:34.5485140Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:43:34.5546710Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-09-07T07:43:34.5547330Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-09-07T07:43:34.5557566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:34.5557800Z env:
2025-09-07T07:43:34.5557954Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:34.5558128Z ##[endgroup]
2025-09-07T07:43:34.5642775Z Prepare all required actions
2025-09-07T07:43:34.5696033Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-09-07T07:43:34.5696255Z with:
2025-09-07T07:43:34.5696806Z   github-token: ***
2025-09-07T07:43:34.5696972Z env:
2025-09-07T07:43:34.5697124Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:34.5697289Z ##[endgroup]
2025-09-07T07:43:34.5795013Z ##[group]Run set -eux
2025-09-07T07:43:34.5795200Z set -eux
2025-09-07T07:43:34.5795478Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-09-07T07:43:34.5803956Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:34.5804195Z env:
2025-09-07T07:43:34.5804344Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:34.5804660Z   GITHUB_TOKEN: ***
2025-09-07T07:43:34.5804824Z ##[endgroup]
2025-09-07T07:43:34.5831864Z + python3 .github/scripts/get_workflow_job_id.py 17525294857 i-0d9447236daff7b96
2025-09-07T07:43:35.0038701Z Setting output job-id=49775530527
2025-09-07T07:43:35.0039257Z Setting output job-name=inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:43:35.0167142Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84
2025-09-07T07:43:35.0167632Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84
2025-09-07T07:43:35.0168208Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 &
2025-09-07T07:43:35.0168714Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2025-09-07T07:43:35.0176784Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:35.0177016Z env:
2025-09-07T07:43:35.0177174Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:35.0177352Z   JOB_ID: 49775530527
2025-09-07T07:43:35.0177745Z   JOB_NAME: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:43:35.0178114Z   WORKFLOW_NAME: inductor-perf-nightly-x86-zen
2025-09-07T07:43:35.0178371Z   WORKFLOW_RUN_ID: 17525294857
2025-09-07T07:43:35.0178561Z   MONITOR_LOG_INTERVAL: 15
2025-09-07T07:43:35.0178730Z   MONITOR_DATA_COLLECT_INTERVAL: 4
2025-09-07T07:43:35.0178921Z ##[endgroup]
2025-09-07T07:43:35.5205831Z Defaulting to user installation because normal site-packages is not writeable
2025-09-07T07:43:35.7730907Z Collecting psutil==5.9.8
2025-09-07T07:43:35.7878549Z   Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
2025-09-07T07:43:35.8580832Z Collecting dataclasses_json==0.6.7
2025-09-07T07:43:35.8606785Z   Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB)
2025-09-07T07:43:35.8869997Z Collecting nvidia-ml-py==11.525.84
2025-09-07T07:43:35.8894079Z   Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB)
2025-09-07T07:43:35.9223429Z Collecting typing-inspect<1,>=0.4.0
2025-09-07T07:43:35.9248686Z   Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
2025-09-07T07:43:36.0009877Z Collecting marshmallow<4.0.0,>=3.18.0
2025-09-07T07:43:36.0073492Z   Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB)
2025-09-07T07:43:36.0608504Z Collecting packaging>=17.0
2025-09-07T07:43:36.0633984Z   Downloading packaging-25.0-py3-none-any.whl (66 kB)
2025-09-07T07:43:36.1137231Z Collecting mypy-extensions>=0.3.0
2025-09-07T07:43:36.1162126Z   Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB)
2025-09-07T07:43:36.1811387Z Collecting typing-extensions>=3.7.4
2025-09-07T07:43:36.1845633Z   Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
2025-09-07T07:43:36.3872802Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json
2025-09-07T07:43:36.9833959Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.15.0 typing-inspect-0.9.0
2025-09-07T07:43:37.2008650Z Prepare all required actions
2025-09-07T07:43:37.2008949Z Getting action download info
2025-09-07T07:43:37.3592676Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-09-07T07:43:38.1815857Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-09-07T07:43:40.6931438Z ##[group]Run ./.github/actions/download-build-artifacts
2025-09-07T07:43:40.6931687Z with:
2025-09-07T07:43:40.6931854Z   name: linux-jammy-py3.9-gcc11-build
2025-09-07T07:43:40.6932053Z   s3-bucket: gha-artifacts
2025-09-07T07:43:40.6932225Z env:
2025-09-07T07:43:40.6932363Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:40.6932525Z ##[endgroup]
2025-09-07T07:43:40.7077397Z ##[group]Run seemethere/download-artifact-s3@v4
2025-09-07T07:43:40.7077616Z with:
2025-09-07T07:43:40.7077778Z   name: linux-jammy-py3.9-gcc11-build
2025-09-07T07:43:40.7077984Z   s3-bucket: gha-artifacts
2025-09-07T07:43:40.7078188Z   region: us-east-1
2025-09-07T07:43:40.7078333Z env:
2025-09-07T07:43:40.7078478Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:40.7078657Z ##[endgroup]
2025-09-07T07:43:41.4753343Z (node:57460) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-09-07T07:43:41.4753666Z 
2025-09-07T07:43:41.4753803Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-09-07T07:43:41.4754153Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-09-07T07:43:41.4754510Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-09-07T07:43:43.0852062Z Found 1 objects with prefix pytorch/pytorch/17525294857/linux-jammy-py3.9-gcc11-build/
2025-09-07T07:43:43.0852575Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-09-07T07:43:47.6751259Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-09-07T07:43:47.6755992Z Artifact download has finished successfully
2025-09-07T07:43:47.7026640Z ##[group]Run unzip -o artifacts.zip
2025-09-07T07:43:47.7026901Z unzip -o artifacts.zip
2025-09-07T07:43:47.7035101Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:47.7035340Z env:
2025-09-07T07:43:47.7035491Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:47.7035667Z ##[endgroup]
2025-09-07T07:43:47.7268995Z Archive:  artifacts.zip
2025-09-07T07:43:47.7269974Z    creating: dist/
2025-09-07T07:43:48.7115641Z   inflating: dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl  
2025-09-07T07:43:48.7223602Z   inflating: dist/.ninja_log         
2025-09-07T07:43:48.7224189Z    creating: build/custom_test_artifacts/
2025-09-07T07:43:48.7225075Z    creating: build/custom_test_artifacts/custom-op-build/
2025-09-07T07:43:48.7225446Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-09-07T07:43:48.7225810Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-09-07T07:43:48.7228921Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-09-07T07:43:48.7229881Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/
2025-09-07T07:43:48.7230281Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-09-07T07:43:48.7230701Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/
2025-09-07T07:43:48.7231114Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-09-07T07:43:48.7233054Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-09-07T07:43:48.7234113Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-09-07T07:43:48.7234815Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-09-07T07:43:48.7235485Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-09-07T07:43:48.7235917Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-09-07T07:43:48.7238385Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-09-07T07:43:48.7239632Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-09-07T07:43:48.7240473Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-09-07T07:43:48.7241809Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-09-07T07:43:48.7243300Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-09-07T07:43:48.7243757Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-09-07T07:43:48.7244184Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-09-07T07:43:48.7244585Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-09-07T07:43:48.7245036Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-09-07T07:43:48.7245529Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-09-07T07:43:48.7246001Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-09-07T07:43:48.7247178Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-09-07T07:43:48.7247748Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-09-07T07:43:48.7248246Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-09-07T07:43:48.7248772Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-09-07T07:43:48.7249250Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-09-07T07:43:48.7249741Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-09-07T07:43:48.7267123Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-09-07T07:43:48.7434883Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-09-07T07:43:48.7435334Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-09-07T07:43:48.7435819Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-09-07T07:43:48.7436356Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-09-07T07:43:48.7436884Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-09-07T07:43:48.7437352Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-09-07T07:43:48.7438312Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-09-07T07:43:48.7438804Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-09-07T07:43:48.7439294Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-09-07T07:43:48.7439778Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-09-07T07:43:48.7440269Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-09-07T07:43:48.7457904Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-09-07T07:43:48.7525206Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-09-07T07:43:48.7525752Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-09-07T07:43:48.7526253Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-09-07T07:43:48.7526707Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-09-07T07:43:48.7527389Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-09-07T07:43:48.7529012Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-09-07T07:43:48.7529442Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json  
2025-09-07T07:43:48.7530831Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-09-07T07:43:48.7531582Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-09-07T07:43:48.7532349Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-09-07T07:43:48.7676766Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-09-07T07:43:48.7723570Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-09-07T07:43:48.7723895Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-09-07T07:43:48.7724196Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-09-07T07:43:48.7724556Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-09-07T07:43:48.7727797Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-09-07T07:43:48.7728188Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/
2025-09-07T07:43:48.7728647Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-09-07T07:43:48.7729082Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/
2025-09-07T07:43:48.7729484Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-09-07T07:43:48.7731534Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-09-07T07:43:48.7732772Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-09-07T07:43:48.7733456Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-09-07T07:43:48.7733882Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-09-07T07:43:48.7734296Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-09-07T07:43:48.7736384Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-09-07T07:43:48.7737742Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-09-07T07:43:48.7739141Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-09-07T07:43:48.7740423Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-09-07T07:43:48.7742225Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-09-07T07:43:48.7742678Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-09-07T07:43:48.7743066Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-09-07T07:43:48.7743465Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-09-07T07:43:48.7744170Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-09-07T07:43:48.7744964Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-09-07T07:43:48.7745468Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-09-07T07:43:48.7745943Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-09-07T07:43:48.7746416Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-09-07T07:43:48.7746895Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-09-07T07:43:48.7747368Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-09-07T07:43:48.7747853Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-09-07T07:43:48.7748466Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-09-07T07:43:48.7765975Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-09-07T07:43:48.7818869Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-09-07T07:43:48.7819501Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-09-07T07:43:48.7820052Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-09-07T07:43:48.7820631Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-09-07T07:43:48.7821269Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-09-07T07:43:48.7822466Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-09-07T07:43:48.7823017Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json  
2025-09-07T07:43:48.7824327Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-09-07T07:43:48.7825093Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-09-07T07:43:48.7825671Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-09-07T07:43:48.7857962Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-09-07T07:43:48.7858291Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-09-07T07:43:48.7858628Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-09-07T07:43:48.7859013Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-09-07T07:43:48.7862212Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-09-07T07:43:48.7862650Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/
2025-09-07T07:43:48.7863082Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-09-07T07:43:48.7863559Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/
2025-09-07T07:43:48.7864005Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-09-07T07:43:48.7865866Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-09-07T07:43:48.7867103Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-09-07T07:43:48.7867758Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-09-07T07:43:48.7868225Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-09-07T07:43:48.7868681Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-09-07T07:43:48.7870586Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-09-07T07:43:48.7871971Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-09-07T07:43:48.7872658Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-09-07T07:43:48.7874043Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-09-07T07:43:48.7875535Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-09-07T07:43:48.7876026Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-09-07T07:43:48.7876451Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-09-07T07:43:48.7876882Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-09-07T07:43:48.7877363Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-09-07T07:43:48.7877926Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-09-07T07:43:48.7878465Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-09-07T07:43:48.7878961Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-09-07T07:43:48.7879480Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-09-07T07:43:48.7880183Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-09-07T07:43:48.7880812Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-09-07T07:43:48.7881345Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-09-07T07:43:48.7882123Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-09-07T07:43:48.7885864Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-09-07T07:43:48.7986404Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-09-07T07:43:48.7986916Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-09-07T07:43:48.7987432Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-09-07T07:43:48.7987998Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-09-07T07:43:48.7988542Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-09-07T07:43:48.7989048Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-09-07T07:43:48.7989584Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-09-07T07:43:48.7990443Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-09-07T07:43:48.7990979Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-09-07T07:43:48.7991513Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-09-07T07:43:48.7992034Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-09-07T07:43:48.8009569Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-09-07T07:43:48.8054812Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-09-07T07:43:48.8055552Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-09-07T07:43:48.8056054Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-09-07T07:43:48.8056516Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-09-07T07:43:48.8057296Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-09-07T07:43:48.8058565Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-09-07T07:43:48.8059006Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json  
2025-09-07T07:43:48.8060309Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-09-07T07:43:48.8061083Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-09-07T07:43:48.8061611Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-09-07T07:43:48.8146733Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-09-07T07:43:48.8179535Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-09-07T07:43:48.8179836Z    creating: build/lib/
2025-09-07T07:43:48.8249291Z   inflating: build/lib/libprotobuf-lite.a  
2025-09-07T07:43:48.8620820Z   inflating: build/lib/libprotobuf.a  
2025-09-07T07:43:48.9036325Z   inflating: build/lib/libprotoc.a   
2025-09-07T07:43:48.9044680Z   inflating: build/lib/libpthreadpool.a  
2025-09-07T07:43:48.9051614Z   inflating: build/lib/libcpuinfo.a  
2025-09-07T07:43:48.9058346Z   inflating: build/lib/libcpuinfo_internals.a  
2025-09-07T07:43:48.9059331Z   inflating: build/lib/libclog.a     
2025-09-07T07:43:48.9075254Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-09-07T07:43:48.9077393Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-09-07T07:43:48.9233141Z   inflating: build/lib/libmicrokernels-prod.a  
2025-09-07T07:43:48.9248263Z   inflating: build/lib/libnnpack.a   
2025-09-07T07:43:48.9984472Z   inflating: build/lib/libmicrokernels-all.a  
2025-09-07T07:43:49.0043601Z   inflating: build/lib/libgtest.a    
2025-09-07T07:43:49.0057885Z   inflating: build/lib/libgmock.a    
2025-09-07T07:43:49.0058610Z   inflating: build/lib/libgtest_main.a  
2025-09-07T07:43:49.0059524Z   inflating: build/lib/libgmock_main.a  
2025-09-07T07:43:49.0136127Z   inflating: build/lib/libXNNPACK.a  
2025-09-07T07:43:49.0200306Z   inflating: build/lib/libbenchmark.a  
2025-09-07T07:43:49.0201021Z   inflating: build/lib/libbenchmark_main.a  
2025-09-07T07:43:49.0201696Z   inflating: build/lib/libjitprofiling.a  
2025-09-07T07:43:49.0208938Z   inflating: build/lib/libittnotify.a  
2025-09-07T07:43:49.0264451Z   inflating: build/lib/libasmjit.a   
2025-09-07T07:43:49.1264928Z   inflating: build/lib/libfbgemm.a   
2025-09-07T07:43:49.1290807Z   inflating: build/lib/libtensorpipe_uv.a  
2025-09-07T07:43:49.1754365Z   inflating: build/lib/libtensorpipe.a  
2025-09-07T07:43:49.1855744Z   inflating: build/lib/libgloo.a     
2025-09-07T07:43:49.1895103Z   inflating: build/lib/libonnx_proto.a  
2025-09-07T07:43:49.2498151Z   inflating: build/lib/libonnx.a     
2025-09-07T07:43:50.1028902Z   inflating: build/lib/libdnnl.a     
2025-09-07T07:43:50.1044112Z   inflating: build/lib/libfmt.a      
2025-09-07T07:43:50.1262876Z   inflating: build/lib/libkineto.a   
2025-09-07T07:43:50.1356444Z   inflating: build/lib/libc10.so     
2025-09-07T07:43:50.1357742Z   inflating: build/lib/libtorch_global_deps.so  
2025-09-07T07:43:52.6387769Z   inflating: build/lib/libtorch_cpu.so  
2025-09-07T07:43:52.6388813Z   inflating: build/lib/libtorch.so   
2025-09-07T07:43:52.6448878Z   inflating: build/lib/libtorchbind_test.so  
2025-09-07T07:43:52.6464473Z   inflating: build/lib/libjitbackend_test.so  
2025-09-07T07:43:52.6484905Z   inflating: build/lib/libbackend_with_compiler.so  
2025-09-07T07:43:52.6507571Z   inflating: build/lib/libaoti_custom_ops.so  
2025-09-07T07:43:52.6511326Z   inflating: build/lib/libshm.so     
2025-09-07T07:43:52.8233799Z   inflating: build/lib/libtorch_python.so  
2025-09-07T07:43:52.8263830Z   inflating: build/lib/libnnapi_backend.so  
2025-09-07T07:43:52.8264690Z    creating: build/bin/
2025-09-07T07:43:52.8264941Z    creating: build/bin/CMakeFiles/
2025-09-07T07:43:52.8265987Z   inflating: build/bin/cmake_install.cmake  
2025-09-07T07:43:52.8266244Z   inflating: build/bin/CTestTestfile.cmake  
2025-09-07T07:43:52.8647957Z   inflating: build/bin/protoc-3.13.0.0  
2025-09-07T07:43:52.9029620Z   inflating: build/bin/protoc        
2025-09-07T07:43:52.9076410Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-09-07T07:43:52.9125832Z   inflating: build/bin/c10_AllocatorConfig_test  
2025-09-07T07:43:52.9174011Z   inflating: build/bin/c10_DeviceGuard_test  
2025-09-07T07:43:52.9222488Z   inflating: build/bin/c10_Device_test  
2025-09-07T07:43:52.9268786Z   inflating: build/bin/c10_StreamGuard_test  
2025-09-07T07:43:52.9323497Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-09-07T07:43:52.9373753Z   inflating: build/bin/c10_Scalar_test  
2025-09-07T07:43:52.9426423Z   inflating: build/bin/c10_SymInt_test  
2025-09-07T07:43:52.9477198Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-09-07T07:43:52.9528499Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-09-07T07:43:52.9580399Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-09-07T07:43:52.9644386Z   inflating: build/bin/c10_cow_test  
2025-09-07T07:43:52.9690789Z   inflating: build/bin/c10_ArrayRef_test  
2025-09-07T07:43:52.9736379Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-09-07T07:43:52.9785995Z   inflating: build/bin/c10_Bitset_test  
2025-09-07T07:43:52.9832305Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-09-07T07:43:52.9884975Z   inflating: build/bin/c10_Enumerate_test  
2025-09-07T07:43:52.9932481Z   inflating: build/bin/c10_Half_test  
2025-09-07T07:43:52.9981383Z   inflating: build/bin/c10_IntrusiveList_test  
2025-09-07T07:43:53.0033140Z   inflating: build/bin/c10_LeftRight_test  
2025-09-07T07:43:53.0084356Z   inflating: build/bin/c10_Metaprogramming_test  
2025-09-07T07:43:53.0133826Z   inflating: build/bin/c10_NetworkFlow_test  
2025-09-07T07:43:53.0180291Z   inflating: build/bin/c10_Synchronized_test  
2025-09-07T07:43:53.0227070Z   inflating: build/bin/c10_Semaphore_test  
2025-09-07T07:43:53.0274972Z   inflating: build/bin/c10_TypeIndex_test  
2025-09-07T07:43:53.0322432Z   inflating: build/bin/c10_TypeList_test  
2025-09-07T07:43:53.0373631Z   inflating: build/bin/c10_ThreadLocal_test  
2025-09-07T07:43:53.0419831Z   inflating: build/bin/c10_TypeTraits_test  
2025-09-07T07:43:53.0467695Z   inflating: build/bin/c10_accumulate_test  
2025-09-07T07:43:53.0520032Z   inflating: build/bin/c10_bfloat16_test  
2025-09-07T07:43:53.0566899Z   inflating: build/bin/c10_bit_cast_test  
2025-09-07T07:43:53.0619629Z   inflating: build/bin/c10_complex_math_test  
2025-09-07T07:43:53.0670783Z   inflating: build/bin/c10_complex_test  
2025-09-07T07:43:53.0716915Z   inflating: build/bin/c10_error_test  
2025-09-07T07:43:53.0765926Z   inflating: build/bin/c10_exception_test  
2025-09-07T07:43:53.0812620Z   inflating: build/bin/c10_flags_test  
2025-09-07T07:43:53.0859940Z   inflating: build/bin/c10_irange_test  
2025-09-07T07:43:53.0906759Z   inflating: build/bin/c10_generic_math_test  
2025-09-07T07:43:53.1050901Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-09-07T07:43:53.1100274Z   inflating: build/bin/c10_lazy_test  
2025-09-07T07:43:53.1153173Z   inflating: build/bin/c10_logging_test  
2025-09-07T07:43:53.1202314Z   inflating: build/bin/c10_registry_test  
2025-09-07T07:43:53.1271117Z   inflating: build/bin/c10_optional_test  
2025-09-07T07:43:53.1327607Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-09-07T07:43:53.1463527Z   inflating: build/bin/c10_small_vector_test  
2025-09-07T07:43:53.1511842Z   inflating: build/bin/c10_ssize_test  
2025-09-07T07:43:53.1564669Z   inflating: build/bin/c10_string_util_test  
2025-09-07T07:43:53.1610097Z   inflating: build/bin/c10_string_view_test  
2025-09-07T07:43:53.1656816Z   inflating: build/bin/c10_tempfile_test  
2025-09-07T07:43:53.1697321Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-09-07T07:43:53.1749504Z   inflating: build/bin/c10_typeid_test  
2025-09-07T07:43:53.2254523Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-09-07T07:43:53.2772609Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-09-07T07:43:53.3297155Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-09-07T07:43:53.3347018Z   inflating: build/bin/static_runtime_bench  
2025-09-07T07:43:53.3564752Z   inflating: build/bin/static_runtime_test  
2025-09-07T07:43:53.3632056Z   inflating: build/bin/Dict_test     
2025-09-07T07:43:53.3680458Z   inflating: build/bin/Dimname_test  
2025-09-07T07:43:53.3740189Z   inflating: build/bin/MaybeOwned_test  
2025-09-07T07:43:53.3792448Z   inflating: build/bin/NamedTensor_test  
2025-09-07T07:43:53.3847490Z   inflating: build/bin/apply_utils_test  
2025-09-07T07:43:53.3901954Z   inflating: build/bin/atest         
2025-09-07T07:43:53.3960598Z   inflating: build/bin/basic         
2025-09-07T07:43:53.4012285Z   inflating: build/bin/broadcast_test  
2025-09-07T07:43:53.4059463Z   inflating: build/bin/cpu_allocator_test  
2025-09-07T07:43:53.4113481Z   inflating: build/bin/cpu_generator_test  
2025-09-07T07:43:53.4162701Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-09-07T07:43:53.4245444Z   inflating: build/bin/cpu_rng_test  
2025-09-07T07:43:53.4292748Z   inflating: build/bin/dlconvertor_test  
2025-09-07T07:43:53.4345501Z   inflating: build/bin/extension_backend_test  
2025-09-07T07:43:53.4396991Z   inflating: build/bin/half_test     
2025-09-07T07:43:53.4482955Z   inflating: build/bin/ivalue_test   
2025-09-07T07:43:53.4529479Z   inflating: build/bin/lazy_tensor_test  
2025-09-07T07:43:53.4578808Z   inflating: build/bin/math_kernel_test  
2025-09-07T07:43:53.4628484Z   inflating: build/bin/memory_format_test  
2025-09-07T07:43:53.4677919Z   inflating: build/bin/memory_overlapping_test  
2025-09-07T07:43:53.4727433Z   inflating: build/bin/mobile_memory_cleanup  
2025-09-07T07:43:53.4779209Z   inflating: build/bin/native_test   
2025-09-07T07:43:53.4827048Z   inflating: build/bin/operator_name_test  
2025-09-07T07:43:53.4874039Z   inflating: build/bin/operators_test  
2025-09-07T07:43:53.4922662Z   inflating: build/bin/packedtensoraccessor_test  
2025-09-07T07:43:53.4984063Z   inflating: build/bin/pow_test      
2025-09-07T07:43:53.5037021Z   inflating: build/bin/quantized_test  
2025-09-07T07:43:53.5083126Z   inflating: build/bin/reduce_ops_test  
2025-09-07T07:43:53.5130698Z   inflating: build/bin/reportMemoryUsage_test  
2025-09-07T07:43:53.5182366Z   inflating: build/bin/scalar_tensor_test  
2025-09-07T07:43:53.5236968Z   inflating: build/bin/scalar_test   
2025-09-07T07:43:53.5284158Z   inflating: build/bin/StorageUtils_test  
2025-09-07T07:43:53.5332891Z   inflating: build/bin/stride_properties_test  
2025-09-07T07:43:53.5404121Z   inflating: build/bin/tensor_iterator_test  
2025-09-07T07:43:53.5454358Z   inflating: build/bin/test_parallel  
2025-09-07T07:43:53.5501269Z   inflating: build/bin/thread_init_test  
2025-09-07T07:43:53.5552521Z   inflating: build/bin/type_ptr_test  
2025-09-07T07:43:53.5607029Z   inflating: build/bin/type_test     
2025-09-07T07:43:53.5656068Z   inflating: build/bin/undefined_tensor_test  
2025-09-07T07:43:53.5702141Z   inflating: build/bin/verify_api_visibility  
2025-09-07T07:43:53.5766314Z   inflating: build/bin/legacy_vmap_test  
2025-09-07T07:43:53.5814258Z   inflating: build/bin/weakref_test  
2025-09-07T07:43:53.5861723Z   inflating: build/bin/wrapdim_test  
2025-09-07T07:43:53.5909583Z   inflating: build/bin/xla_tensor_test  
2025-09-07T07:43:53.5964229Z   inflating: build/bin/IListRef_test  
2025-09-07T07:43:53.6058463Z   inflating: build/bin/List_test     
2025-09-07T07:43:53.6118937Z   inflating: build/bin/KernelFunction_test  
2025-09-07T07:43:53.6226208Z   inflating: build/bin/kernel_function_legacy_test  
2025-09-07T07:43:53.6310952Z   inflating: build/bin/kernel_function_test  
2025-09-07T07:43:53.6422428Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-09-07T07:43:53.6513315Z   inflating: build/bin/kernel_lambda_test  
2025-09-07T07:43:53.6568700Z   inflating: build/bin/kernel_stackbased_test  
2025-09-07T07:43:53.6654066Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-09-07T07:43:53.6701404Z   inflating: build/bin/CppSignature_test  
2025-09-07T07:43:53.6752433Z   inflating: build/bin/backend_fallback_test  
2025-09-07T07:43:53.6798186Z   inflating: build/bin/op_allowlist_test  
2025-09-07T07:43:53.7067781Z   inflating: build/bin/op_registration_test  
2025-09-07T07:43:53.7129026Z   inflating: build/bin/inline_container_test  
2025-09-07T07:43:53.8071628Z   inflating: build/bin/test_jit      
2025-09-07T07:43:53.8121101Z   inflating: build/bin/FileStoreTest  
2025-09-07T07:43:53.8169971Z   inflating: build/bin/BackoffTest   
2025-09-07T07:43:53.8222812Z   inflating: build/bin/TCPStoreTest  
2025-09-07T07:43:53.8545491Z   inflating: build/bin/test_nativert  
2025-09-07T07:43:53.8595362Z   inflating: build/bin/HashStoreTest  
2025-09-07T07:43:53.8655769Z   inflating: build/bin/ProcessGroupGlooTest  
2025-09-07T07:43:53.8658378Z   inflating: build/bin/example_allreduce  
2025-09-07T07:43:53.8709153Z   inflating: build/bin/test_dist_autograd  
2025-09-07T07:43:53.8770489Z   inflating: build/bin/test_cpp_rpc  
2025-09-07T07:43:53.9738642Z   inflating: build/bin/test_api      
2025-09-07T07:43:53.9740900Z   inflating: build/bin/parallel_benchmark  
2025-09-07T07:43:54.0037477Z   inflating: build/bin/test_lazy     
2025-09-07T07:43:54.0041241Z   inflating: build/bin/torch_shm_manager  
2025-09-07T07:43:54.0041514Z    creating: .additional_ci_files/
2025-09-07T07:43:54.0116668Z   inflating: .additional_ci_files/test-times.json  
2025-09-07T07:43:54.0401364Z   inflating: .additional_ci_files/test-class-times.json  
2025-09-07T07:43:54.0458858Z ##[group]Run rm artifacts.zip
2025-09-07T07:43:54.0459084Z rm artifacts.zip
2025-09-07T07:43:54.0467452Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:54.0467726Z env:
2025-09-07T07:43:54.0467876Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:54.0468054Z ##[endgroup]
2025-09-07T07:43:54.1605480Z ##[group]Run df -H
2025-09-07T07:43:54.1605668Z df -H
2025-09-07T07:43:54.1612860Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:54.1613093Z env:
2025-09-07T07:43:54.1613242Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:54.1613417Z ##[endgroup]
2025-09-07T07:43:54.1662460Z Filesystem        Size  Used Avail Use% Mounted on
2025-09-07T07:43:54.1662745Z devtmpfs          4.2M     0  4.2M   0% /dev
2025-09-07T07:43:54.1662957Z tmpfs             199G     0  199G   0% /dev/shm
2025-09-07T07:43:54.1663188Z tmpfs              80G  1.3M   80G   1% /run
2025-09-07T07:43:54.1663412Z /dev/nvme0n1p1    215G   72G  144G  34% /
2025-09-07T07:43:54.1663658Z tmpfs             199G   13k  199G   1% /tmp
2025-09-07T07:43:54.1663877Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2025-09-07T07:43:54.1690969Z Prepare all required actions
2025-09-07T07:43:54.1691772Z Getting action download info
2025-09-07T07:43:54.2975574Z ##[group]Run ./.github/actions/download-td-artifacts
2025-09-07T07:43:54.2975815Z with:
2025-09-07T07:43:54.2975956Z env:
2025-09-07T07:43:54.2976098Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:54.2976263Z ##[endgroup]
2025-09-07T07:43:54.3018905Z ##[group]Run seemethere/download-artifact-s3@v4
2025-09-07T07:43:54.3019121Z with:
2025-09-07T07:43:54.3019253Z   name: td_results
2025-09-07T07:43:54.3019410Z   s3-bucket: gha-artifacts
2025-09-07T07:43:54.3019585Z   region: us-east-1
2025-09-07T07:43:54.3019733Z env:
2025-09-07T07:43:54.3019870Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:54.3020044Z ##[endgroup]
2025-09-07T07:43:54.6641449Z (node:57483) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-09-07T07:43:54.6641771Z 
2025-09-07T07:43:54.6641956Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-09-07T07:43:54.6642321Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-09-07T07:43:54.7758730Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-09-07T07:43:54.7759093Z Found 0 objects with prefix pytorch/pytorch/17525294857/td_results/
2025-09-07T07:43:54.7763842Z Artifact download has finished successfully
2025-09-07T07:43:54.8048972Z ##[group]Run mkdir -p .additional_ci_files
2025-09-07T07:43:54.8049228Z mkdir -p .additional_ci_files
2025-09-07T07:43:54.8049513Z mv td_results.json .additional_ci_files/td_results.json || true
2025-09-07T07:43:54.8056768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:54.8057017Z env:
2025-09-07T07:43:54.8057166Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:54.8057344Z ##[endgroup]
2025-09-07T07:43:54.8115333Z mv: cannot stat 'td_results.json': No such file or directory
2025-09-07T07:43:54.8169773Z ##[group]Run .github/scripts/parse_ref.py
2025-09-07T07:43:54.8170072Z .github/scripts/parse_ref.py
2025-09-07T07:43:54.8177778Z shell: /usr/bin/bash -e {0}
2025-09-07T07:43:54.8177983Z env:
2025-09-07T07:43:54.8178149Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:54.8178348Z ##[endgroup]
2025-09-07T07:43:54.8787564Z Setting output branch=main
2025-09-07T07:43:54.8873918Z Prepare all required actions
2025-09-07T07:43:54.8874202Z Getting action download info
2025-09-07T07:43:55.0067656Z ##[group]Run ./.github/actions/filter-test-configs
2025-09-07T07:43:55.0067887Z with:
2025-09-07T07:43:55.0068190Z   github-token: ***
2025-09-07T07:43:55.0070624Z   test-matrix: {"include": [{"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 1, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 2, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 3, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 1, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 2, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 3, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 4, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 5, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 1, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 2, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 3, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 4, "num_shards": 4, "runner": "linux.24xlarge.amd"}]}
2025-09-07T07:43:55.0073354Z   job-name: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:43:55.0073863Z env:
2025-09-07T07:43:55.0074006Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:55.0074182Z ##[endgroup]
2025-09-07T07:43:55.0156955Z ##[group]Run nick-fields/retry@v3.0.0
2025-09-07T07:43:55.0157153Z with:
2025-09-07T07:43:55.0157290Z   shell: bash
2025-09-07T07:43:55.0157440Z   timeout_minutes: 10
2025-09-07T07:43:55.0157600Z   max_attempts: 5
2025-09-07T07:43:55.0157765Z   retry_wait_seconds: 30
2025-09-07T07:43:55.0158239Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.2

2025-09-07T07:43:55.0158723Z   polling_interval_seconds: 1
2025-09-07T07:43:55.0158910Z   warning_on_retry: true
2025-09-07T07:43:55.0159081Z   continue_on_error: false
2025-09-07T07:43:55.0159247Z env:
2025-09-07T07:43:55.0159376Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:55.0159667Z   GITHUB_TOKEN: ***
2025-09-07T07:43:55.0159834Z ##[endgroup]
2025-09-07T07:43:55.1659204Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2
2025-09-07T07:43:55.3386952Z Defaulting to user installation because normal site-packages is not writeable
2025-09-07T07:43:55.5760451Z Collecting requests==2.27.1
2025-09-07T07:43:55.6035743Z   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
2025-09-07T07:43:55.8390556Z Collecting pyyaml==6.0.2
2025-09-07T07:43:55.8471216Z   Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB)
2025-09-07T07:43:55.9571820Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2025-09-07T07:43:56.2272095Z Collecting charset-normalizer~=2.0.0
2025-09-07T07:43:56.2352964Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2025-09-07T07:43:56.3609668Z Collecting certifi>=2017.4.17
2025-09-07T07:43:56.3688753Z   Downloading certifi-2025.8.3-py3-none-any.whl (161 kB)
2025-09-07T07:43:56.4180646Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2025-09-07T07:43:56.4727515Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml
2025-09-07T07:43:56.8209474Z Successfully installed certifi-2025.8.3 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1
2025-09-07T07:43:57.0763536Z Command completed after 1 attempt(s).
2025-09-07T07:43:57.0824059Z ##[group]Run set -x
2025-09-07T07:43:57.0824254Z set -x
2025-09-07T07:43:57.0824401Z 
2025-09-07T07:43:57.0824641Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-09-07T07:43:57.0824936Z # in runner workspace
2025-09-07T07:43:57.0825189Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-09-07T07:43:57.0832411Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:57.0832648Z env:
2025-09-07T07:43:57.0832791Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:57.0832966Z ##[endgroup]
2025-09-07T07:43:57.0860106Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-09-07T07:43:57.1008068Z Setting output branch=main
2025-09-07T07:43:57.1051482Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-09-07T07:43:57.1051764Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-09-07T07:43:57.1051978Z echo "Job name: ${JOB_NAME}"
2025-09-07T07:43:57.1052160Z 
2025-09-07T07:43:57.1052427Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-09-07T07:43:57.1052730Z # in runner workspace
2025-09-07T07:43:57.1052996Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-09-07T07:43:57.1053299Z   --workflow "${GITHUB_WORKFLOW}" \
2025-09-07T07:43:57.1053508Z   --job-name "${JOB_NAME}" \
2025-09-07T07:43:57.1055954Z   --test-matrix "{"include": [{"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 1, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 2, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 3, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 1, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 2, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 3, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 4, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 5, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 1, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 2, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 3, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 4, "num_shards": 4, "runner": "linux.24xlarge.amd"}]}" \
2025-09-07T07:43:57.1058770Z   --selected-test-configs "" \
2025-09-07T07:43:57.1058977Z   --pr-number "${PR_NUMBER}" \
2025-09-07T07:43:57.1059167Z   --tag "${TAG}" \
2025-09-07T07:43:57.1059352Z   --event-name "${EVENT_NAME}" \
2025-09-07T07:43:57.1059558Z   --schedule "${SCHEDULE}" \
2025-09-07T07:43:57.1059743Z   --branch "${HEAD_BRANCH}"
2025-09-07T07:43:57.1066820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:57.1067060Z env:
2025-09-07T07:43:57.1067205Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:57.1067615Z   GITHUB_TOKEN: ***
2025-09-07T07:43:57.1067932Z   JOB_NAME: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:43:57.1068276Z   PR_NUMBER: 
2025-09-07T07:43:57.1068423Z   TAG: 
2025-09-07T07:43:57.1068561Z   EVENT_NAME: schedule
2025-09-07T07:43:57.1068715Z   SCHEDULE: 0 7 * * *
2025-09-07T07:43:57.1068874Z   HEAD_BRANCH: main
2025-09-07T07:43:57.1069185Z ##[endgroup]
2025-09-07T07:43:57.1096230Z Workflow: inductor-perf-nightly-x86-zen
2025-09-07T07:43:57.1096623Z Job name: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:43:57.2585898Z Setting output keep-going=True
2025-09-07T07:43:57.2586172Z Setting output ci-verbose-test-logs=False
2025-09-07T07:43:57.2586408Z Setting output ci-test-showlocals=False
2025-09-07T07:43:57.2586624Z Setting output ci-no-test-timeout=False
2025-09-07T07:43:57.2586834Z Setting output ci-no-td=False
2025-09-07T07:43:57.2587026Z Setting output ci-td-distributed=False
2025-09-07T07:43:57.2587238Z Setting output is-unstable=False
2025-09-07T07:43:57.2587436Z Setting output reenabled-issues=
2025-09-07T07:43:57.2589959Z Setting output test-matrix={"include": [{"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 1, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 2, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 3, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 1, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 2, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 3, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 4, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 5, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 1, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 2, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 3, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 4, "num_shards": 4, "runner": "linux.24xlarge.amd"}]}
2025-09-07T07:43:57.2593019Z Setting output is-test-matrix-empty=False
2025-09-07T07:43:57.2705620Z ##[group]Run echo "Filtered matrix:"
2025-09-07T07:43:57.2705865Z echo "Filtered matrix:"
2025-09-07T07:43:57.2708274Z echo "{"include": [{"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 1, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 2, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_huggingface_perf_cpu_x86_zen", "shard": 3, "num_shards": 3, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 1, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 2, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 3, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 4, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_timm_perf_cpu_x86_zen", "shard": 5, "num_shards": 5, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 1, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 2, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 3, "num_shards": 4, "runner": "linux.24xlarge.amd"}, {"config": "inductor_torchbench_perf_cpu_x86_zen", "shard": 4, "num_shards": 4, "runner": "linux.24xlarge.amd"}]}"
2025-09-07T07:43:57.2710867Z 
2025-09-07T07:43:57.2711001Z echo
2025-09-07T07:43:57.2711187Z echo "Is the current job unstable? False"
2025-09-07T07:43:57.2711395Z 
2025-09-07T07:43:57.2711526Z echo
2025-09-07T07:43:57.2711685Z echo "Is keep-going label set? True"
2025-09-07T07:43:57.2712081Z 
2025-09-07T07:43:57.2712214Z echo
2025-09-07T07:43:57.2712363Z echo "Reenabled issues? "
2025-09-07T07:43:57.2719538Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:57.2719778Z env:
2025-09-07T07:43:57.2719928Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:57.2720095Z ##[endgroup]
2025-09-07T07:43:57.2746370Z Filtered matrix:
2025-09-07T07:43:57.2749375Z {include: [{config: inductor_huggingface_perf_cpu_x86_zen, shard: 1, num_shards: 3, runner: linux.24xlarge.amd}, {config: inductor_huggingface_perf_cpu_x86_zen, shard: 2, num_shards: 3, runner: linux.24xlarge.amd}, {config: inductor_huggingface_perf_cpu_x86_zen, shard: 3, num_shards: 3, runner: linux.24xlarge.amd}, {config: inductor_timm_perf_cpu_x86_zen, shard: 1, num_shards: 5, runner: linux.24xlarge.amd}, {config: inductor_timm_perf_cpu_x86_zen, shard: 2, num_shards: 5, runner: linux.24xlarge.amd}, {config: inductor_timm_perf_cpu_x86_zen, shard: 3, num_shards: 5, runner: linux.24xlarge.amd}, {config: inductor_timm_perf_cpu_x86_zen, shard: 4, num_shards: 5, runner: linux.24xlarge.amd}, {config: inductor_timm_perf_cpu_x86_zen, shard: 5, num_shards: 5, runner: linux.24xlarge.amd}, {config: inductor_torchbench_perf_cpu_x86_zen, shard: 1, num_shards: 4, runner: linux.24xlarge.amd}, {config: inductor_torchbench_perf_cpu_x86_zen, shard: 2, num_shards: 4, runner: linux.24xlarge.amd}, {config: inductor_torchbench_perf_cpu_x86_zen, shard: 3, num_shards: 4, runner: linux.24xlarge.amd}, {config: inductor_torchbench_perf_cpu_x86_zen, shard: 4, num_shards: 4, runner: linux.24xlarge.amd}]}
2025-09-07T07:43:57.2751933Z 
2025-09-07T07:43:57.2752014Z Is the current job unstable? False
2025-09-07T07:43:57.2752158Z 
2025-09-07T07:43:57.2752240Z Is keep-going label set? True
2025-09-07T07:43:57.2752642Z 
2025-09-07T07:43:57.2752715Z Reenabled issues? 
2025-09-07T07:43:57.2825595Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-09-07T07:43:57.2825954Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-09-07T07:43:57.2833038Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:57.2833285Z env:
2025-09-07T07:43:57.2833439Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:57.2833619Z   JOB_TIMEOUT: 720
2025-09-07T07:43:57.2833767Z ##[endgroup]
2025-09-07T07:43:57.2971933Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-09-07T07:43:57.2972282Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-09-07T07:43:57.2972558Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-09-07T07:43:57.2979724Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T07:43:57.2979969Z env:
2025-09-07T07:43:57.2980118Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:57.2980310Z ##[endgroup]
2025-09-07T07:43:57.3138168Z ##[group]Run set -x
2025-09-07T07:43:57.3138417Z set -x
2025-09-07T07:43:57.3138555Z 
2025-09-07T07:43:57.3138725Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-09-07T07:43:57.3138980Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-09-07T07:43:57.3139236Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-09-07T07:43:57.3139454Z   TEST_COMMAND=.ci/onnx/test.sh
2025-09-07T07:43:57.3139643Z else
2025-09-07T07:43:57.3139821Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-09-07T07:43:57.3140017Z fi
2025-09-07T07:43:57.3140153Z 
2025-09-07T07:43:57.3140329Z # Leaving 1GB for the runner and other things
2025-09-07T07:43:57.3140686Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo)
2025-09-07T07:43:57.3141236Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap
2025-09-07T07:43:57.3141666Z # comes from https://github.com/pytorch/test-infra/pull/6058
2025-09-07T07:43:57.3141995Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3))
2025-09-07T07:43:57.3142245Z 
2025-09-07T07:43:57.3142420Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then
2025-09-07T07:43:57.3142628Z   SHM_OPTS=
2025-09-07T07:43:57.3142792Z   JENKINS_USER=
2025-09-07T07:43:57.3143018Z   # ensure that docker container cleanly exits in 12 hours
2025-09-07T07:43:57.3143306Z   # if for some reason cleanup action doesn't stop container
2025-09-07T07:43:57.3143549Z   # when job is cancelled
2025-09-07T07:43:57.3143740Z   DOCKER_SHELL_CMD="sleep 12h"
2025-09-07T07:43:57.3143924Z else
2025-09-07T07:43:57.3144089Z   SHM_OPTS="--shm-size=${SHM_SIZE}"
2025-09-07T07:43:57.3144295Z   JENKINS_USER="--user jenkins"
2025-09-07T07:43:57.3144493Z   DOCKER_SHELL_CMD=
2025-09-07T07:43:57.3144664Z fi
2025-09-07T07:43:57.3144798Z 
2025-09-07T07:43:57.3145008Z # detached container should get cleaned up by teardown_ec2_linux
2025-09-07T07:43:57.3145321Z # TODO: Stop building test binaries as part of the build phase
2025-09-07T07:43:57.3145684Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice
2025-09-07T07:43:57.3145998Z # shellcheck disable=SC2086,SC2090
2025-09-07T07:43:57.3146208Z container_name=$(docker run \
2025-09-07T07:43:57.3146404Z   ${GPU_FLAG:-} \
2025-09-07T07:43:57.3146596Z   ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \
2025-09-07T07:43:57.3146812Z   -e BUILD_ENVIRONMENT \
2025-09-07T07:43:57.3146996Z   -e PR_NUMBER \
2025-09-07T07:43:57.3147169Z   -e GITHUB_ACTIONS \
2025-09-07T07:43:57.3147345Z   -e GITHUB_REPOSITORY \
2025-09-07T07:43:57.3147692Z   -e GITHUB_WORKFLOW \
2025-09-07T07:43:57.3147873Z   -e GITHUB_JOB \
2025-09-07T07:43:57.3148056Z   -e GITHUB_RUN_ID \
2025-09-07T07:43:57.3148224Z   -e GITHUB_RUN_NUMBER \
2025-09-07T07:43:57.3148409Z   -e GITHUB_RUN_ATTEMPT \
2025-09-07T07:43:57.3148599Z   -e JOB_ID \
2025-09-07T07:43:57.3148799Z   -e JOB_NAME \
2025-09-07T07:43:57.3148989Z   -e BASE_SHA \
2025-09-07T07:43:57.3149176Z   -e BRANCH \
2025-09-07T07:43:57.3149341Z   -e SHA1 \
2025-09-07T07:43:57.3149519Z   -e AWS_DEFAULT_REGION \
2025-09-07T07:43:57.3149714Z   -e IN_WHEEL_TEST \
2025-09-07T07:43:57.3149913Z   -e SHARD_NUMBER \
2025-09-07T07:43:57.3150093Z   -e TEST_CONFIG \
2025-09-07T07:43:57.3150271Z   -e NUM_TEST_SHARDS \
2025-09-07T07:43:57.3150450Z   -e REENABLED_ISSUES \
2025-09-07T07:43:57.3150644Z   -e CONTINUE_THROUGH_ERROR \
2025-09-07T07:43:57.3150960Z   -e VERBOSE_TEST_LOGS \
2025-09-07T07:43:57.3151149Z   -e TEST_SHOWLOCALS \
2025-09-07T07:43:57.3151321Z   -e NO_TEST_TIMEOUT \
2025-09-07T07:43:57.3151494Z   -e NO_TD \
2025-09-07T07:43:57.3151662Z   -e TD_DISTRIBUTED \
2025-09-07T07:43:57.3151842Z   -e PR_LABELS \
2025-09-07T07:43:57.3152034Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-09-07T07:43:57.3152242Z   -e SCCACHE_BUCKET \
2025-09-07T07:43:57.3152415Z   -e SCCACHE_REGION \
2025-09-07T07:43:57.3152584Z   -e XLA_CUDA \
2025-09-07T07:43:57.3152758Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2025-09-07T07:43:57.3152979Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-09-07T07:43:57.3153208Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-09-07T07:43:57.3153436Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2025-09-07T07:43:57.3153647Z   -e HUGGING_FACE_HUB_TOKEN \
2025-09-07T07:43:57.3153854Z   -e VLLM_TEST_HUGGING_FACE_TOKEN \
2025-09-07T07:43:57.3154069Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2025-09-07T07:43:57.3154274Z   -e DASHBOARD_TAG \
2025-09-07T07:43:57.3154457Z   -e ARTIFACTS_FILE_SUFFIX \
2025-09-07T07:43:57.3154681Z   --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \
2025-09-07T07:43:57.3154941Z   --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \
2025-09-07T07:43:57.3155195Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2025-09-07T07:43:57.3155432Z   --security-opt seccomp=unconfined \
2025-09-07T07:43:57.3155647Z   --cap-add=SYS_PTRACE \
2025-09-07T07:43:57.3155833Z   --ipc=host \
2025-09-07T07:43:57.3168420Z   ${SHM_OPTS} \
2025-09-07T07:43:57.3168610Z   --tty \
2025-09-07T07:43:57.3168766Z   --detach \
2025-09-07T07:43:57.3168938Z   --name="${container_name}" \
2025-09-07T07:43:57.3169147Z   ${JENKINS_USER} \
2025-09-07T07:43:57.3169392Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-09-07T07:43:57.3169644Z   -w /var/lib/jenkins/workspace \
2025-09-07T07:43:57.3169844Z   "${DOCKER_IMAGE}" \
2025-09-07T07:43:57.3170019Z   ${DOCKER_SHELL_CMD}
2025-09-07T07:43:57.3170183Z )
2025-09-07T07:43:57.3170365Z # Propagate download.pytorch.org IP to container
2025-09-07T07:43:57.3170768Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2025-09-07T07:43:57.3171194Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2025-09-07T07:43:57.3171445Z 
2025-09-07T07:43:57.3171624Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then
2025-09-07T07:43:57.3171968Z   docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt"
2025-09-07T07:43:57.3172273Z fi
2025-09-07T07:43:57.3172513Z 
2025-09-07T07:43:57.3172813Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2025-09-07T07:43:57.3180015Z shell: /usr/bin/bash -e {0}
2025-09-07T07:43:57.3180189Z env:
2025-09-07T07:43:57.3180335Z   GIT_DEFAULT_BRANCH: main
2025-09-07T07:43:57.3180550Z   BUILD_ENVIRONMENT: linux-jammy-py3.9-gcc11-build
2025-09-07T07:43:57.3180768Z   PR_NUMBER: 
2025-09-07T07:43:57.3180924Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-09-07T07:43:57.3181152Z   GITHUB_WORKFLOW: inductor-perf-nightly-x86-zen
2025-09-07T07:43:57.3181358Z   GITHUB_JOB: test
2025-09-07T07:43:57.3181509Z   GITHUB_RUN_ID: 17525294857
2025-09-07T07:43:57.3181672Z   GITHUB_RUN_NUMBER: 91
2025-09-07T07:43:57.3181831Z   GITHUB_RUN_ATTEMPT: 1
2025-09-07T07:43:57.3181982Z   JOB_ID: 49775530527
2025-09-07T07:43:57.3182298Z   JOB_NAME: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:43:57.3182623Z   BRANCH: main
2025-09-07T07:43:57.3182916Z   SHA1: 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:43:57.3183158Z   BASE_SHA: 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:43:57.3183402Z   TEST_CONFIG: inductor_torchbench_perf_cpu_x86_zen
2025-09-07T07:43:57.3183604Z   SHARD_NUMBER: 2
2025-09-07T07:43:57.3183752Z   NUM_TEST_SHARDS: 4
2025-09-07T07:43:57.3183897Z   REENABLED_ISSUES: 
2025-09-07T07:43:57.3184056Z   CONTINUE_THROUGH_ERROR: True
2025-09-07T07:43:57.3184231Z   VERBOSE_TEST_LOGS: False
2025-09-07T07:43:57.3184418Z   TEST_SHOWLOCALS: False
2025-09-07T07:43:57.3184594Z   NO_TEST_TIMEOUT: False
2025-09-07T07:43:57.3184745Z   NO_TD: False
2025-09-07T07:43:57.3184883Z   TD_DISTRIBUTED: False
2025-09-07T07:43:57.3185081Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2025-09-07T07:43:57.3185301Z   SCCACHE_REGION: us-east-1
2025-09-07T07:43:57.3185467Z   SHM_SIZE: 1g
2025-09-07T07:43:57.3185974Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:43:57.3186492Z   XLA_CUDA: 
2025-09-07T07:43:57.3186714Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2025-09-07T07:43:57.3186992Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-09-07T07:43:57.3187191Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-09-07T07:43:57.3187592Z   DASHBOARD_TAG: training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true
2025-09-07T07:43:57.3188163Z   VLLM_TEST_HUGGING_FACE_TOKEN: ***
2025-09-07T07:43:57.3188422Z   HUGGING_FACE_HUB_TOKEN: ***
2025-09-07T07:43:57.3188682Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: ***
2025-09-07T07:43:57.3189021Z   ARTIFACTS_FILE_SUFFIX: test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T07:43:57.3189343Z ##[endgroup]
2025-09-07T07:43:57.3214352Z + [[ inductor_torchbench_perf_cpu_x86_zen == \m\u\l\t\i\g\p\u ]]
2025-09-07T07:43:57.3214641Z + [[ linux-jammy-py3.9-gcc11-build == *onnx* ]]
2025-09-07T07:43:57.3214881Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-09-07T07:43:57.3217985Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo
2025-09-07T07:43:57.3239526Z + TOTAL_AVAILABLE_MEMORY_IN_GB='368.765 '
2025-09-07T07:43:57.3239873Z + TOTAL_MEMORY_WITH_SWAP=371
2025-09-07T07:43:57.3240129Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]]
2025-09-07T07:43:57.3240358Z + SHM_OPTS=--shm-size=1g
2025-09-07T07:43:57.3240534Z + JENKINS_USER='--user jenkins'
2025-09-07T07:43:57.3240720Z + DOCKER_SHELL_CMD=
2025-09-07T07:43:57.3249090Z +++ nproc --ignore=2
2025-09-07T07:43:57.3681524Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=94 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e VLLM_TEST_HUGGING_FACE_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=368g --memory-swap=371g --env-file=/tmp/github_env_17525294857 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T07:46:45.6681589Z + container_name=de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T07:46:45.6686807Z + grep download.pytorch.org /etc/hosts
2025-09-07T07:46:45.6693537Z + docker exec -i de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac sudo bash -c '/bin/cat >> /etc/hosts'
2025-09-07T07:46:45.7899897Z + echo DOCKER_CONTAINER_ID=de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T07:46:45.7900793Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]]
2025-09-07T07:46:45.7908922Z ++ echo dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl
2025-09-07T07:46:45.7911229Z + docker exec -t de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac sh -c 'python3 -m pip install dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2025-09-07T07:46:46.1455453Z Processing ./dist/torch-2.9.0a0+git93fb23d-cp39-cp39-linux_x86_64.whl (from torch==2.9.0a0+git93fb23d)
2025-09-07T07:46:46.3412438Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.19.1)
2025-09-07T07:46:46.3414174Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (4.15.0)
2025-09-07T07:46:46.3416881Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (1.13.3)
2025-09-07T07:46:46.3419882Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (2.8.8)
2025-09-07T07:46:46.3421995Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.1.6)
2025-09-07T07:46:46.3424743Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (2025.3.0)
2025-09-07T07:46:46.3433884Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.3.0)
2025-09-07T07:46:46.3699183Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opt-einsum>=3.3->torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (1.22.4)
2025-09-07T07:46:46.3712876Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (1.3.0)
2025-09-07T07:46:46.3741215Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from jinja2->torch==2.9.0a0+git93fb23d->torch==2.9.0a0+git93fb23d) (3.0.2)
2025-09-07T07:46:47.0723122Z Installing collected packages: torch
2025-09-07T07:46:54.1385645Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-09-07T07:46:54.1386913Z dall-e 0.1 requires torchvision, which is not installed.
2025-09-07T07:46:54.1387203Z effdet 0.4.1 requires torchvision, which is not installed.
2025-09-07T07:46:54.1387554Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed.
2025-09-07T07:46:54.1388008Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed.
2025-09-07T07:46:54.1388482Z timm 1.0.14 requires torchvision, which is not installed.
2025-09-07T07:46:54.1388816Z Successfully installed torch-2.9.0a0+git93fb23d
2025-09-07T07:46:54.2175973Z + export TERM=vt100
2025-09-07T07:46:54.2176178Z + TERM=vt100
2025-09-07T07:46:54.2178831Z ++ dirname .ci/pytorch/test.sh
2025-09-07T07:46:54.2189684Z + source .ci/pytorch/common.sh
2025-09-07T07:46:54.2193434Z +++ dirname .ci/pytorch/common.sh
2025-09-07T07:46:54.2202713Z ++ source .ci/pytorch/common_utils.sh
2025-09-07T07:46:54.2204543Z +++ declare -f -t trap_add
2025-09-07T07:46:54.2208187Z ++ set -ex -o pipefail
2025-09-07T07:46:54.2208407Z ++ [[ linux-jammy-py3.9-gcc11-build == *rocm* ]]
2025-09-07T07:46:54.2208635Z ++ BUILD_TEST_LIBTORCH=0
2025-09-07T07:46:54.2212587Z ++ dirname .ci/pytorch/test.sh
2025-09-07T07:46:54.2222400Z + source .ci/pytorch/common-build.sh
2025-09-07T07:46:54.2223811Z ++ [[ linux-jammy-py3.9-gcc11-build != *win-* ]]
2025-09-07T07:46:54.2231204Z ++++ dirname .ci/pytorch/common-build.sh
2025-09-07T07:46:54.2240032Z +++ cd .ci/pytorch
2025-09-07T07:46:54.2240425Z +++ pwd -P
2025-09-07T07:46:54.2242899Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch
2025-09-07T07:46:54.2243213Z ++ [[ linux-jammy-py3.9-gcc11-build == *-pch* ]]
2025-09-07T07:46:54.2243428Z ++ which sccache
2025-09-07T07:46:54.2264731Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]]
2025-09-07T07:46:54.2264951Z ++ sccache --stop-server
2025-09-07T07:46:54.2295815Z ++ true
2025-09-07T07:46:54.2295981Z ++ rm -f /var/lib/jenkins/sccache_error.log
2025-09-07T07:46:54.2309666Z ++ trap_add sccache_epilogue EXIT
2025-09-07T07:46:54.2309924Z ++ trap_add_cmd=sccache_epilogue
2025-09-07T07:46:54.2310118Z ++ shift
2025-09-07T07:46:54.2310275Z ++ for trap_add_name in "$@"
2025-09-07T07:46:54.2315748Z ++++ trap -p EXIT
2025-09-07T07:46:54.2318373Z +++ eval 'extract_trap_cmd '
2025-09-07T07:46:54.2318565Z ++++ extract_trap_cmd
2025-09-07T07:46:54.2318727Z ++++ printf '%s\n' ''
2025-09-07T07:46:54.2318933Z +++ printf '%s\n' sccache_epilogue
2025-09-07T07:46:54.2320759Z ++ trap -- '
2025-09-07T07:46:54.2320914Z sccache_epilogue' EXIT
2025-09-07T07:46:54.2321221Z ++ [[ -n 1 ]]
2025-09-07T07:46:54.2321508Z ++ echo 'Skipping sccache server initialization, setting environment variables'
2025-09-07T07:46:54.2321883Z Skipping sccache server initialization, setting environment variables
2025-09-07T07:46:54.2322168Z ++ export SCCACHE_IDLE_TIMEOUT=0
2025-09-07T07:46:54.2322360Z ++ SCCACHE_IDLE_TIMEOUT=0
2025-09-07T07:46:54.2323841Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-09-07T07:46:54.2324141Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-09-07T07:46:54.2324414Z ++ export RUST_LOG=sccache::server=error
2025-09-07T07:46:54.2324625Z ++ RUST_LOG=sccache::server=error
2025-09-07T07:46:54.2324812Z ++ sccache --zero-stats
2025-09-07T07:46:54.4064855Z Statistics zeroed.
2025-09-07T07:46:54.4074254Z ++ which ccache
2025-09-07T07:46:54.4097743Z + [[ linux-jammy-py3.9-gcc11-build != *rocm* ]]
2025-09-07T07:46:54.4098003Z + [[ linux-jammy-py3.9-gcc11-build != *s390x* ]]
2025-09-07T07:46:54.4098234Z + [[ -d /var/lib/jenkins/workspace ]]
2025-09-07T07:46:54.4101605Z ++ stat -c %u /var/lib/jenkins/workspace
2025-09-07T07:46:54.4115977Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2025-09-07T07:46:54.4116187Z + trap_add cleanup_workspace EXIT
2025-09-07T07:46:54.4116384Z + trap_add_cmd=cleanup_workspace
2025-09-07T07:46:54.4116559Z + shift
2025-09-07T07:46:54.4116697Z + for trap_add_name in "$@"
2025-09-07T07:46:54.4126060Z +++ trap -p EXIT
2025-09-07T07:46:54.4127429Z ++ eval 'extract_trap_cmd trap -- '\''
2025-09-07T07:46:54.4127976Z sccache_epilogue'\'' EXIT'
2025-09-07T07:46:54.4128149Z +++ extract_trap_cmd trap -- '
2025-09-07T07:46:54.4128333Z sccache_epilogue' EXIT
2025-09-07T07:46:54.4128494Z +++ printf '%s\n' '
2025-09-07T07:46:54.4128646Z sccache_epilogue'
2025-09-07T07:46:54.4128822Z ++ printf '%s\n' cleanup_workspace
2025-09-07T07:46:54.4129957Z + trap -- '
2025-09-07T07:46:54.4130107Z sccache_epilogue
2025-09-07T07:46:54.4130261Z cleanup_workspace' EXIT
2025-09-07T07:46:54.4130457Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2025-09-07T07:46:55.2222778Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2025-09-07T07:46:55.2243490Z + echo 'Environment variables:'
2025-09-07T07:46:55.2243687Z Environment variables:
2025-09-07T07:46:55.2243860Z + env
2025-09-07T07:46:55.2254752Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-09-07T07:46:55.2255068Z CONTINUE_THROUGH_ERROR=True
2025-09-07T07:46:55.2255289Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build
2025-09-07T07:46:55.2256121Z VLLM_TEST_HUGGING_FACE_TOKEN=***
2025-09-07T07:46:55.2256316Z HOSTNAME=de0caba1838c
2025-09-07T07:46:55.2256681Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2257055Z GITHUB_ACTION=__run_2
2025-09-07T07:46:55.2257232Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-09-07T07:46:55.2257425Z GITHUB_RUN_NUMBER=91
2025-09-07T07:46:55.2257611Z TEST_CONFIG=inductor_torchbench_perf_cpu_x86_zen
2025-09-07T07:46:55.2257886Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-09-07T07:46:55.2258099Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2025-09-07T07:46:55.2258302Z SCCACHE_IDLE_TIMEOUT=0
2025-09-07T07:46:55.2258552Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2025-09-07T07:46:55.2258744Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2025-09-07T07:46:55.2258938Z GITHUB_REF_TYPE=branch
2025-09-07T07:46:55.2259125Z BASE_SHA=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2259330Z XLA_CUDA=
2025-09-07T07:46:55.2259484Z NCCL_LIB_DIR=/usr/local/cuda/lib64/
2025-09-07T07:46:55.2259775Z HUGGING_FACE_HUB_TOKEN=***
2025-09-07T07:46:55.2260118Z ***
2025-09-07T07:46:55.2260267Z GITHUB_REPOSITORY_ID=65600975
2025-09-07T07:46:55.2260446Z GITHUB_ACTIONS=true
2025-09-07T07:46:55.2260642Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-09-07T07:46:55.2260890Z SHA1=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2261122Z GITHUB_SHA=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2261584Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly-x86-zen.yml@refs/heads/main
2025-09-07T07:46:55.2261951Z UCC_HOME=/usr
2025-09-07T07:46:55.2262093Z VERBOSE_TEST_LOGS=False
2025-09-07T07:46:55.2262266Z GITHUB_REF=refs/heads/main
2025-09-07T07:46:55.2262436Z SHARD_NUMBER=2
2025-09-07T07:46:55.2262588Z GITHUB_REF_PROTECTED=true
2025-09-07T07:46:55.2262750Z HOME=/var/lib/jenkins
2025-09-07T07:46:55.2262937Z GITHUB_API_URL=https://api.github.com
2025-09-07T07:46:55.2263167Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-09-07T07:46:55.2263357Z UCX_COMMIT=
2025-09-07T07:46:55.2263492Z USE_SYSTEM_NCCL=1
2025-09-07T07:46:55.2263642Z NUM_TEST_SHARDS=4
2025-09-07T07:46:55.2263786Z UCX_HOME=/usr
2025-09-07T07:46:55.2264136Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2264667Z JOB_NAME=inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:46:55.2265178Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2265648Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2025-09-07T07:46:55.2265943Z GITHUB_EVENT_NAME=schedule
2025-09-07T07:46:55.2266334Z DASHBOARD_TAG=training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true
2025-09-07T07:46:55.2266740Z GITHUB_RUN_ID=17525294857
2025-09-07T07:46:55.2267094Z INSTALLED_OPENBLAS=
2025-09-07T07:46:55.2267466Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2267865Z GITHUB_ACTOR=pytorchmergebot
2025-09-07T07:46:55.2268044Z PR_NUMBER=
2025-09-07T07:46:55.2268173Z DESIRED_CUDA=
2025-09-07T07:46:55.2268320Z GITHUB_RUN_ATTEMPT=1
2025-09-07T07:46:55.2268489Z ANACONDA_PYTHON_VERSION=3.9
2025-09-07T07:46:55.2268700Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-09-07T07:46:55.2268908Z TERM=vt100
2025-09-07T07:46:55.2269049Z INSTALLED_VISION=yes
2025-09-07T07:46:55.2269204Z BRANCH=main
2025-09-07T07:46:55.2269350Z SCCACHE_REGION=us-east-1
2025-09-07T07:46:55.2269526Z OPENSSL_ROOT_DIR=/opt/openssl
2025-09-07T07:46:55.2269710Z CUDA_PATH=/usr/local/cuda
2025-09-07T07:46:55.2270028Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2025-09-07T07:46:55.2270370Z GITHUB_SERVER_URL=https://github.com
2025-09-07T07:46:55.2270563Z UCC_COMMIT=
2025-09-07T07:46:55.2270789Z REENABLED_ISSUES=
2025-09-07T07:46:55.2270935Z DOCS=yes
2025-09-07T07:46:55.2271071Z SHLVL=1
2025-09-07T07:46:55.2271198Z MAX_JOBS=94
2025-09-07T07:46:55.2271343Z GITHUB_ACTOR_ID=97764156
2025-09-07T07:46:55.2271558Z GITHUB_WORKFLOW_SHA=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2271794Z GITHUB_REF_NAME=main
2025-09-07T07:46:55.2272028Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2025-09-07T07:46:55.2272288Z GITHUB_JOB=test
2025-09-07T07:46:55.2272441Z NO_TEST_TIMEOUT=False
2025-09-07T07:46:55.2272600Z TD_DISTRIBUTED=False
2025-09-07T07:46:55.2272767Z GITHUB_REPOSITORY=pytorch/pytorch
2025-09-07T07:46:55.2272956Z GITHUB_RETENTION_DAYS=90
2025-09-07T07:46:55.2273126Z OPENSSL_DIR=/opt/openssl
2025-09-07T07:46:55.2273289Z GITHUB_ACTION_REPOSITORY=
2025-09-07T07:46:55.2273764Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-09-07T07:46:55.2274260Z GITHUB_BASE_REF=
2025-09-07T07:46:55.2274420Z INSTALLED_ACL=
2025-09-07T07:46:55.2274717Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T07:46:55.2275030Z CI=true
2025-09-07T07:46:55.2275179Z GITHUB_REPOSITORY_OWNER=pytorch
2025-09-07T07:46:55.2275405Z RUST_LOG=sccache::server=error
2025-09-07T07:46:55.2275582Z JOB_ID=49775530527
2025-09-07T07:46:55.2275725Z GITHUB_HEAD_REF=
2025-09-07T07:46:55.2275872Z GITHUB_ACTION_REF=
2025-09-07T07:46:55.2276060Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2025-09-07T07:46:55.2276279Z TEST_SHOWLOCALS=False
2025-09-07T07:46:55.2276467Z GITHUB_WORKFLOW=inductor-perf-nightly-x86-zen
2025-09-07T07:46:55.2276691Z DEBIAN_FRONTEND=noninteractive
2025-09-07T07:46:55.2277070Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2277443Z NO_TD=False
2025-09-07T07:46:55.2277600Z SKIP_SCCACHE_INITIALIZATION=1
2025-09-07T07:46:55.2277794Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/
2025-09-07T07:46:55.2277988Z _=/usr/bin/env
2025-09-07T07:46:55.2278207Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-09-07T07:46:55.2515059Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch
2025-09-07T07:46:55.2515477Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin
2025-09-07T07:46:55.2515823Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib
2025-09-07T07:46:55.2516165Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/test
2025-09-07T07:46:55.2516429Z + BUILD_DIR=build
2025-09-07T07:46:55.2516590Z + BUILD_RENAMED_DIR=build_renamed
2025-09-07T07:46:55.2516787Z + BUILD_BIN_DIR=build/bin
2025-09-07T07:46:55.2516952Z + SHARD_NUMBER=2
2025-09-07T07:46:55.2517099Z + NUM_TEST_SHARDS=4
2025-09-07T07:46:55.2517262Z + export TORCH_SERIALIZATION_DEBUG=1
2025-09-07T07:46:55.2517674Z + TORCH_SERIALIZATION_DEBUG=1
2025-09-07T07:46:55.2517863Z + export VALGRIND=ON
2025-09-07T07:46:55.2518015Z + VALGRIND=ON
2025-09-07T07:46:55.2518194Z + [[ linux-jammy-py3.9-gcc11-build == *clang9* ]]
2025-09-07T07:46:55.2518435Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]]
2025-09-07T07:46:55.2518640Z + detect_cuda_arch
2025-09-07T07:46:55.2518815Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]]
2025-09-07T07:46:55.2519037Z + [[ linux-jammy-py3.9-gcc11-build == *s390x* ]]
2025-09-07T07:46:55.2519235Z + [[ 0 == \1 ]]
2025-09-07T07:46:55.2519375Z + [[ True == \1 ]]
2025-09-07T07:46:55.2519542Z + [[ linux-jammy-py3.9-gcc11-build != *bazel* ]]
2025-09-07T07:46:55.2520227Z ++ realpath build/custom_test_artifacts
2025-09-07T07:46:55.2532281Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2025-09-07T07:46:55.2533003Z + [[ -n '' ]]
2025-09-07T07:46:55.2533167Z + echo 'Environment variables'
2025-09-07T07:46:55.2533348Z Environment variables
2025-09-07T07:46:55.2533508Z + env
2025-09-07T07:46:55.2553084Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-09-07T07:46:55.2553391Z CONTINUE_THROUGH_ERROR=True
2025-09-07T07:46:55.2553608Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build
2025-09-07T07:46:55.2553990Z VLLM_TEST_HUGGING_FACE_TOKEN=***
2025-09-07T07:46:55.2554176Z HOSTNAME=de0caba1838c
2025-09-07T07:46:55.2554531Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2554898Z GITHUB_ACTION=__run_2
2025-09-07T07:46:55.2555072Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-09-07T07:46:55.2555262Z GITHUB_RUN_NUMBER=91
2025-09-07T07:46:55.2555444Z TEST_CONFIG=inductor_torchbench_perf_cpu_x86_zen
2025-09-07T07:46:55.2555669Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-09-07T07:46:55.2555877Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2025-09-07T07:46:55.2556072Z SCCACHE_IDLE_TIMEOUT=0
2025-09-07T07:46:55.2556323Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2025-09-07T07:46:55.2556520Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2025-09-07T07:46:55.2556718Z GITHUB_REF_TYPE=branch
2025-09-07T07:46:55.2556902Z BASE_SHA=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2557099Z XLA_CUDA=
2025-09-07T07:46:55.2557248Z NCCL_LIB_DIR=/usr/local/cuda/lib64/
2025-09-07T07:46:55.2557574Z HUGGING_FACE_HUB_TOKEN=***
2025-09-07T07:46:55.2557803Z ***
2025-09-07T07:46:55.2557952Z GITHUB_REPOSITORY_ID=65600975
2025-09-07T07:46:55.2558128Z GITHUB_ACTIONS=true
2025-09-07T07:46:55.2558319Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-09-07T07:46:55.2558567Z SHA1=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2558795Z GITHUB_SHA=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2559195Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly-x86-zen.yml@refs/heads/main
2025-09-07T07:46:55.2559554Z UCC_HOME=/usr
2025-09-07T07:46:55.2559706Z TORCH_SERIALIZATION_DEBUG=1
2025-09-07T07:46:55.2559880Z VERBOSE_TEST_LOGS=False
2025-09-07T07:46:55.2560042Z GITHUB_REF=refs/heads/main
2025-09-07T07:46:55.2560206Z SHARD_NUMBER=2
2025-09-07T07:46:55.2560357Z GITHUB_REF_PROTECTED=true
2025-09-07T07:46:55.2560520Z HOME=/var/lib/jenkins
2025-09-07T07:46:55.2560694Z GITHUB_API_URL=https://api.github.com
2025-09-07T07:46:55.2560914Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-09-07T07:46:55.2561100Z UCX_COMMIT=
2025-09-07T07:46:55.2561240Z USE_SYSTEM_NCCL=1
2025-09-07T07:46:55.2561382Z NUM_TEST_SHARDS=4
2025-09-07T07:46:55.2561526Z UCX_HOME=/usr
2025-09-07T07:46:55.2561883Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2562407Z JOB_NAME=inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T07:46:55.2562903Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2563367Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2025-09-07T07:46:55.2563760Z GITHUB_EVENT_NAME=schedule
2025-09-07T07:46:55.2564155Z DASHBOARD_TAG=training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true
2025-09-07T07:46:55.2564560Z GITHUB_RUN_ID=17525294857
2025-09-07T07:46:55.2564770Z INSTALLED_OPENBLAS=
2025-09-07T07:46:55.2565127Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2565522Z GITHUB_ACTOR=pytorchmergebot
2025-09-07T07:46:55.2565694Z PR_NUMBER=
2025-09-07T07:46:55.2565834Z DESIRED_CUDA=
2025-09-07T07:46:55.2565970Z GITHUB_RUN_ATTEMPT=1
2025-09-07T07:46:55.2566121Z VALGRIND=ON
2025-09-07T07:46:55.2566269Z ANACONDA_PYTHON_VERSION=3.9
2025-09-07T07:46:55.2566478Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-09-07T07:46:55.2566684Z TERM=vt100
2025-09-07T07:46:55.2566828Z INSTALLED_VISION=yes
2025-09-07T07:46:55.2566981Z BRANCH=main
2025-09-07T07:46:55.2567121Z SCCACHE_REGION=us-east-1
2025-09-07T07:46:55.2567379Z OPENSSL_ROOT_DIR=/opt/openssl
2025-09-07T07:46:55.2567561Z CUDA_PATH=/usr/local/cuda
2025-09-07T07:46:55.2567879Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2025-09-07T07:46:55.2568223Z GITHUB_SERVER_URL=https://github.com
2025-09-07T07:46:55.2568405Z UCC_COMMIT=
2025-09-07T07:46:55.2568545Z REENABLED_ISSUES=
2025-09-07T07:46:55.2568690Z DOCS=yes
2025-09-07T07:46:55.2568820Z SHLVL=1
2025-09-07T07:46:55.2568943Z MAX_JOBS=94
2025-09-07T07:46:55.2569080Z GITHUB_ACTOR_ID=97764156
2025-09-07T07:46:55.2569290Z GITHUB_WORKFLOW_SHA=93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T07:46:55.2569520Z GITHUB_REF_NAME=main
2025-09-07T07:46:55.2569752Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2025-09-07T07:46:55.2570011Z GITHUB_JOB=test
2025-09-07T07:46:55.2570163Z NO_TEST_TIMEOUT=False
2025-09-07T07:46:55.2570322Z TD_DISTRIBUTED=False
2025-09-07T07:46:55.2570484Z GITHUB_REPOSITORY=pytorch/pytorch
2025-09-07T07:46:55.2570680Z GITHUB_RETENTION_DAYS=90
2025-09-07T07:46:55.2570845Z OPENSSL_DIR=/opt/openssl
2025-09-07T07:46:55.2571000Z GITHUB_ACTION_REPOSITORY=
2025-09-07T07:46:55.2571457Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-09-07T07:46:55.2571915Z GITHUB_BASE_REF=
2025-09-07T07:46:55.2572063Z INSTALLED_ACL=
2025-09-07T07:46:55.2572359Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T07:46:55.2572670Z CI=true
2025-09-07T07:46:55.2572818Z GITHUB_REPOSITORY_OWNER=pytorch
2025-09-07T07:46:55.2573035Z RUST_LOG=sccache::server=error
2025-09-07T07:46:55.2573209Z JOB_ID=49775530527
2025-09-07T07:46:55.2573352Z GITHUB_HEAD_REF=
2025-09-07T07:46:55.2573497Z GITHUB_ACTION_REF=
2025-09-07T07:46:55.2573682Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2025-09-07T07:46:55.2573898Z TEST_SHOWLOCALS=False
2025-09-07T07:46:55.2574092Z GITHUB_WORKFLOW=inductor-perf-nightly-x86-zen
2025-09-07T07:46:55.2574309Z DEBIAN_FRONTEND=noninteractive
2025-09-07T07:46:55.2574680Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_82e2c6f3-282a-484f-91e3-31f2b15ab45b
2025-09-07T07:46:55.2575050Z NO_TD=False
2025-09-07T07:46:55.2575195Z SKIP_SCCACHE_INITIALIZATION=1
2025-09-07T07:46:55.2575387Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/
2025-09-07T07:46:55.2575581Z _=/usr/bin/env
2025-09-07T07:46:55.2575728Z + echo 'Testing pytorch'
2025-09-07T07:46:55.2575888Z Testing pytorch
2025-09-07T07:46:55.2576055Z + export LANG=C.UTF-8
2025-09-07T07:46:55.2576210Z + LANG=C.UTF-8
2025-09-07T07:46:55.2576345Z + PR_NUMBER=
2025-09-07T07:46:55.2576551Z + [[ inductor_torchbench_perf_cpu_x86_zen == \d\e\f\a\u\l\t ]]
2025-09-07T07:46:55.2576848Z + [[ inductor_torchbench_perf_cpu_x86_zen == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-09-07T07:46:55.2577131Z + [[ inductor_torchbench_perf_cpu_x86_zen == \s\l\o\w ]]
2025-09-07T07:46:55.2577522Z + [[ linux-jammy-py3.9-gcc11-build == *slow-gradcheck* ]]
2025-09-07T07:46:55.2577813Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]]
2025-09-07T07:46:55.2578031Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]]
2025-09-07T07:46:55.2578249Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]]
2025-09-07T07:46:55.2578490Z + [[ inductor_torchbench_perf_cpu_x86_zen == *crossref* ]]
2025-09-07T07:46:55.2578724Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]]
2025-09-07T07:46:55.2578936Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]]
2025-09-07T07:46:55.2579161Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]]
2025-09-07T07:46:55.2579380Z + pip_install ninja==1.10.2
2025-09-07T07:46:55.2579608Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-09-07T07:46:55.2579890Z + python3 -m pip install --progress-bar off ninja==1.10.2
2025-09-07T07:46:55.6392898Z Collecting ninja==1.10.2
2025-09-07T07:46:55.6469398Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-09-07T07:46:55.6564004Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-09-07T07:46:56.3377600Z Installing collected packages: ninja
2025-09-07T07:46:56.3377920Z   Attempting uninstall: ninja
2025-09-07T07:46:56.3385316Z     Found existing installation: ninja 1.11.1.3
2025-09-07T07:46:56.3406848Z     Uninstalling ninja-1.11.1.3:
2025-09-07T07:46:56.3478356Z       Successfully uninstalled ninja-1.11.1.3
2025-09-07T07:46:56.4054755Z Successfully installed ninja-1.10.2
2025-09-07T07:46:56.4803903Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-09-07T07:46:56.4804840Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-09-07T07:46:56.4805470Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]]
2025-09-07T07:46:56.4805732Z + [[ linux-jammy-py3.9-gcc11-build == *asan* ]]
2025-09-07T07:46:56.4805962Z + [[ linux-jammy-py3.9-gcc11-build == *-debug* ]]
2025-09-07T07:46:56.4806201Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]]
2025-09-07T07:46:56.4806538Z + echo 'We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass'
2025-09-07T07:46:56.4806943Z We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass
2025-09-07T07:46:56.4809522Z + cd test
2025-09-07T07:46:56.4809754Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-09-07T07:46:56.7616638Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:46:56.7617590Z   import pynvml  # type: ignore[import]
2025-09-07T07:46:57.5846049Z + [[ inductor_torchbench_perf_cpu_x86_zen == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-09-07T07:46:57.5846446Z + [[ inductor_torchbench_perf_cpu_x86_zen == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-09-07T07:46:57.5846815Z + [[ inductor_torchbench_perf_cpu_x86_zen == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]]
2025-09-07T07:46:57.5847965Z + DYNAMO_BENCHMARK_FLAGS=()
2025-09-07T07:46:57.5849668Z + [[ inductor_torchbench_perf_cpu_x86_zen == *pr_time_benchmarks* ]]
2025-09-07T07:46:57.5849996Z + [[ inductor_torchbench_perf_cpu_x86_zen == *dynamo_eager* ]]
2025-09-07T07:46:57.5850272Z + [[ inductor_torchbench_perf_cpu_x86_zen == *aot_eager* ]]
2025-09-07T07:46:57.5850538Z + [[ inductor_torchbench_perf_cpu_x86_zen == *aot_inductor* ]]
2025-09-07T07:46:57.5850832Z + [[ inductor_torchbench_perf_cpu_x86_zen == *max_autotune_inductor* ]]
2025-09-07T07:46:57.5851130Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor* ]]
2025-09-07T07:46:57.5851388Z + [[ inductor_torchbench_perf_cpu_x86_zen != *perf* ]]
2025-09-07T07:46:57.5852100Z + [[ inductor_torchbench_perf_cpu_x86_zen == *dynamic* ]]
2025-09-07T07:46:57.5852339Z + [[ inductor_torchbench_perf_cpu_x86_zen == *cpu* ]]
2025-09-07T07:46:57.5852579Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu)
2025-09-07T07:46:57.6083233Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]]
2025-09-07T07:46:57.6083501Z + [[ linux-jammy-py3.9-gcc11-build == *-bazel-* ]]
2025-09-07T07:46:57.6086559Z + cd test
2025-09-07T07:46:57.6087338Z + python -c 'import torch; print(torch.__config__.show())'
2025-09-07T07:46:57.8845389Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:46:57.8846225Z   import pynvml  # type: ignore[import]
2025-09-07T07:46:58.5109505Z PyTorch built with:
2025-09-07T07:46:58.5109781Z   - GCC 11.4
2025-09-07T07:46:58.5110421Z   - C++ Version: 201703
2025-09-07T07:46:58.5110788Z   - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-09-07T07:46:58.5111230Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-09-07T07:46:58.5111508Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-09-07T07:46:58.5111730Z   - LAPACK is enabled (usually provided by MKL)
2025-09-07T07:46:58.5111929Z   - NNPACK is enabled
2025-09-07T07:46:58.5112102Z   - CPU capability usage: AVX512
2025-09-07T07:46:58.5114869Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=93fb23d6fae7c4e82c4239a1033e522088742634, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 
2025-09-07T07:46:58.5117616Z 
2025-09-07T07:46:58.6954765Z + cd test
2025-09-07T07:46:58.6955037Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-09-07T07:46:58.9679541Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:46:58.9680381Z   import pynvml  # type: ignore[import]
2025-09-07T07:46:59.5971477Z ATen/Parallel:
2025-09-07T07:46:59.5971754Z 	at::get_num_threads() : 96
2025-09-07T07:46:59.5971958Z 	at::get_num_interop_threads() : 96
2025-09-07T07:46:59.5972168Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-09-07T07:46:59.5972362Z 	omp_get_max_threads() : 96
2025-09-07T07:46:59.5972711Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-09-07T07:46:59.5973064Z 	mkl_get_max_threads() : 96
2025-09-07T07:46:59.5973315Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-09-07T07:46:59.5973593Z std::thread::hardware_concurrency() : 96
2025-09-07T07:46:59.5973797Z Environment variables:
2025-09-07T07:46:59.5973975Z 	OMP_NUM_THREADS : [not set]
2025-09-07T07:46:59.5974609Z 	MKL_NUM_THREADS : [not set]
2025-09-07T07:46:59.5974789Z ATen parallel backend: OpenMP
2025-09-07T07:46:59.5974906Z 
2025-09-07T07:46:59.7787712Z + [[ inductor_torchbench_perf_cpu_x86_zen == *numpy_2* ]]
2025-09-07T07:46:59.7788051Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]]
2025-09-07T07:46:59.7788326Z + [[ inductor_torchbench_perf_cpu_x86_zen == *backward* ]]
2025-09-07T07:46:59.7788593Z + [[ inductor_torchbench_perf_cpu_x86_zen == *xla* ]]
2025-09-07T07:46:59.7788832Z + [[ inductor_torchbench_perf_cpu_x86_zen == *vllm* ]]
2025-09-07T07:46:59.7789090Z + [[ inductor_torchbench_perf_cpu_x86_zen == *executorch* ]]
2025-09-07T07:46:59.7789393Z + [[ inductor_torchbench_perf_cpu_x86_zen == \j\i\t\_\l\e\g\a\c\y ]]
2025-09-07T07:46:59.7789676Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]]
2025-09-07T07:46:59.7789928Z + [[ inductor_torchbench_perf_cpu_x86_zen == distributed ]]
2025-09-07T07:46:59.7790219Z + [[ inductor_torchbench_perf_cpu_x86_zen == *operator_benchmark* ]]
2025-09-07T07:46:59.7790939Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor_distributed* ]]
2025-09-07T07:46:59.7791253Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor-halide* ]]
2025-09-07T07:46:59.7791560Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor-triton-cpu* ]]
2025-09-07T07:46:59.7791881Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor-micro-benchmark* ]]
2025-09-07T07:46:59.7792189Z + [[ inductor_torchbench_perf_cpu_x86_zen == *huggingface* ]]
2025-09-07T07:46:59.7792453Z + [[ inductor_torchbench_perf_cpu_x86_zen == *timm* ]]
2025-09-07T07:46:59.7792707Z + [[ inductor_torchbench_perf_cpu_x86_zen == cachebench ]]
2025-09-07T07:46:59.7792967Z + [[ inductor_torchbench_perf_cpu_x86_zen == verify_cachebench ]]
2025-09-07T07:46:59.7793245Z + [[ inductor_torchbench_perf_cpu_x86_zen == *torchbench* ]]
2025-09-07T07:46:59.7793469Z + install_torchaudio
2025-09-07T07:46:59.7793634Z + local commit
2025-09-07T07:46:59.7793791Z ++ get_pinned_commit audio
2025-09-07T07:46:59.7793985Z ++ cat .github/ci_commit_pins/audio.txt
2025-09-07T07:46:59.8132696Z + commit=2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:46:59.8133144Z + pip_build_and_install git+https://github.com/pytorch/audio.git@2e300559e4e123928a22187b8f59a5b56f57ddc8 dist/audio
2025-09-07T07:46:59.8134129Z + local build_target=git+https://github.com/pytorch/audio.git@2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:46:59.8134457Z + local wheel_dir=dist/audio
2025-09-07T07:46:59.8134634Z + local found_whl=0
2025-09-07T07:46:59.8134802Z + for file in "${wheel_dir}"/*.whl
2025-09-07T07:46:59.8135000Z + [[ -f dist/audio/*.whl ]]
2025-09-07T07:46:59.8135165Z + '[' 0 == 0 ']'
2025-09-07T07:46:59.8135613Z + python3 -m pip wheel --no-build-isolation --no-deps --no-use-pep517 -w dist/audio git+https://github.com/pytorch/audio.git@2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:47:00.1037714Z Collecting git+https://github.com/pytorch/audio.git@2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:47:00.1040994Z   Cloning https://github.com/pytorch/audio.git (to revision 2e300559e4e123928a22187b8f59a5b56f57ddc8) to /tmp/pip-req-build-6hlkg3l_
2025-09-07T07:47:00.1928971Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-6hlkg3l_
2025-09-07T07:47:02.7461600Z   Running command git rev-parse -q --verify 'sha^2e300559e4e123928a22187b8f59a5b56f57ddc8'
2025-09-07T07:47:02.7505540Z   Running command git fetch -q https://github.com/pytorch/audio.git 2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:47:02.8609228Z   Running command git checkout -q 2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:47:03.2290538Z   Resolved https://github.com/pytorch/audio.git to commit 2e300559e4e123928a22187b8f59a5b56f57ddc8
2025-09-07T07:47:03.2290984Z   Running command git submodule update --init --recursive -q
2025-09-07T07:47:04.6178398Z   Preparing metadata (setup.py) ... [?25l- \ done
2025-09-07T07:47:04.6216702Z [?25hBuilding wheels for collected packages: torchaudio
2025-09-07T07:47:04.6323532Z   DEPRECATION: Building 'torchaudio' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. A possible replacement is to use the standardized build interface by setting the `--use-pep517` option, (possibly combined with `--no-build-isolation`), or adding a `pyproject.toml` file to the source tree of 'torchaudio'. Discussion can be found at https://github.com/pypa/pip/issues/6334
2025-09-07T07:47:26.4696591Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2025-09-07T07:47:26.4710682Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl size=491187 sha256=546ff9170a0d92d409fcd0637612dab05fc5407ccdb6fcf76c21b3ed9755e7dc
2025-09-07T07:47:26.4712166Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/18/53/66/85c241150a0c0641633cdbd6ae534ee172017679f2ef448df1
2025-09-07T07:47:26.4742266Z Successfully built torchaudio
2025-09-07T07:47:26.5920733Z + for file in "${wheel_dir}"/*.whl
2025-09-07T07:47:26.5921095Z + pip_install_whl dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:26.5921513Z + args=('dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl')
2025-09-07T07:47:26.5921780Z + local args
2025-09-07T07:47:26.5922025Z + [[ dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl == *\ * ]]
2025-09-07T07:47:26.5922328Z + for path in "${args[@]}"
2025-09-07T07:47:26.5922611Z + echo 'Installing dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl'
2025-09-07T07:47:26.5923006Z Installing dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:26.5923447Z + python3 -mpip install --no-index --no-deps dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:26.8920096Z Processing ./dist/audio/torchaudio-2.8.0a0+2e30055-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:26.8969110Z Installing collected packages: torchaudio
2025-09-07T07:47:27.0569075Z Successfully installed torchaudio-2.8.0a0+2e30055
2025-09-07T07:47:27.0877394Z + install_torchvision
2025-09-07T07:47:27.0877591Z + local orig_preload
2025-09-07T07:47:27.0877750Z + local commit
2025-09-07T07:47:27.0882143Z ++ get_pinned_commit vision
2025-09-07T07:47:27.0882341Z ++ cat .github/ci_commit_pins/vision.txt
2025-09-07T07:47:27.0899156Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:27.0899373Z + orig_preload=
2025-09-07T07:47:27.0899529Z + '[' -n '' ']'
2025-09-07T07:47:27.0899712Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]]
2025-09-07T07:47:27.0900154Z + pip_build_and_install git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 dist/vision
2025-09-07T07:47:27.0900682Z + local build_target=git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:27.0901083Z + local wheel_dir=dist/vision
2025-09-07T07:47:27.0901276Z + local found_whl=0
2025-09-07T07:47:27.0901440Z + for file in "${wheel_dir}"/*.whl
2025-09-07T07:47:27.0901631Z + [[ -f dist/vision/*.whl ]]
2025-09-07T07:47:27.0901807Z + '[' 0 == 0 ']'
2025-09-07T07:47:27.0902256Z + python3 -m pip wheel --no-build-isolation --no-deps --no-use-pep517 -w dist/vision git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:27.3830180Z Collecting git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:27.3833979Z   Cloning https://github.com/pytorch/vision.git (to revision 966da7e46f65d6d49df3e31214470a4fe5cc8e66) to /tmp/pip-req-build-75k5m33q
2025-09-07T07:47:27.3885071Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-75k5m33q
2025-09-07T07:47:28.7561794Z   Running command git rev-parse -q --verify 'sha^966da7e46f65d6d49df3e31214470a4fe5cc8e66'
2025-09-07T07:47:28.7606517Z   Running command git fetch -q https://github.com/pytorch/vision.git 966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:28.9147958Z   Running command git checkout -q 966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:29.2015697Z   Resolved https://github.com/pytorch/vision.git to commit 966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-09-07T07:47:30.6911342Z   Preparing metadata (setup.py) ... [?25l- \ | / done
2025-09-07T07:47:30.6951240Z [?25hBuilding wheels for collected packages: torchvision
2025-09-07T07:47:30.7059994Z   DEPRECATION: Building 'torchvision' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. A possible replacement is to use the standardized build interface by setting the `--use-pep517` option, (possibly combined with `--no-build-isolation`), or adding a `pyproject.toml` file to the source tree of 'torchvision'. Discussion can be found at https://github.com/pypa/pip/issues/6334
2025-09-07T07:47:59.2299825Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2025-09-07T07:47:59.2322588Z [?25h  Created wheel for torchvision: filename=torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl size=1301751 sha256=9573c154c3503860798ceb40afb0069782afb98584a9dfd18ea8c8720804076c
2025-09-07T07:47:59.2323377Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/33/6d/2f/9f3e65c401a351a98a00d9d72c4434fdbd3e10256b2d832157
2025-09-07T07:47:59.2361977Z Successfully built torchvision
2025-09-07T07:47:59.3252004Z + for file in "${wheel_dir}"/*.whl
2025-09-07T07:47:59.3252471Z + pip_install_whl dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:59.3264545Z + args=('dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl')
2025-09-07T07:47:59.3264855Z + local args
2025-09-07T07:47:59.3265125Z + [[ dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl == *\ * ]]
2025-09-07T07:47:59.3265434Z + for path in "${args[@]}"
2025-09-07T07:47:59.3265722Z + echo 'Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl'
2025-09-07T07:47:59.3266114Z Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:59.3266566Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:59.6258729Z Processing ./dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl
2025-09-07T07:47:59.6329770Z Installing collected packages: torchvision
2025-09-07T07:48:00.0013950Z Successfully installed torchvision-0.22.0a0+966da7e
2025-09-07T07:48:00.0353989Z + '[' -n '' ']'
2025-09-07T07:48:00.0354175Z + id=1
2025-09-07T07:48:00.0354350Z + pip_install opencv-python==4.8.0.74
2025-09-07T07:48:00.0354629Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-09-07T07:48:00.0354970Z + python3 -m pip install --progress-bar off opencv-python==4.8.0.74
2025-09-07T07:48:00.3989887Z Collecting opencv-python==4.8.0.74
2025-09-07T07:48:00.4230357Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2025-09-07T07:48:00.4287234Z Requirement already satisfied: numpy>=1.17.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opencv-python==4.8.0.74) (1.22.4)
2025-09-07T07:48:00.4395421Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2025-09-07T07:48:01.4921811Z Installing collected packages: opencv-python
2025-09-07T07:48:01.4922084Z   Attempting uninstall: opencv-python
2025-09-07T07:48:01.4932792Z     Found existing installation: opencv-python 4.11.0.86
2025-09-07T07:48:01.5010694Z     Uninstalling opencv-python-4.11.0.86:
2025-09-07T07:48:01.6203704Z       Successfully uninstalled opencv-python-4.11.0.86
2025-09-07T07:48:02.3538752Z Successfully installed opencv-python-4.8.0.74
2025-09-07T07:48:02.4433084Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor_torchbench_smoketest_perf* ]]
2025-09-07T07:48:02.4433528Z + [[ inductor_torchbench_perf_cpu_x86_zen == *inductor_torchbench_cpu_smoketest_perf* ]]
2025-09-07T07:48:02.4433910Z + [[ inductor_torchbench_perf_cpu_x86_zen == *torchbench_gcp_smoketest* ]]
2025-09-07T07:48:02.4434217Z + [[ inductor_torchbench_perf_cpu_x86_zen != *cpu* ]]
2025-09-07T07:48:02.4434449Z + PYTHONPATH=/torchbench
2025-09-07T07:48:02.4434652Z + test_dynamo_benchmark torchbench 1
2025-09-07T07:48:02.4437596Z ++ pwd
2025-09-07T07:48:02.4440491Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2025-09-07T07:48:02.4440766Z + local suite=torchbench
2025-09-07T07:48:02.4440933Z + shift
2025-09-07T07:48:02.4441066Z + local shard_id=1
2025-09-07T07:48:02.4441221Z + shift
2025-09-07T07:48:02.4441425Z + [[ inductor_torchbench_perf_cpu_x86_zen == *perf_compare* ]]
2025-09-07T07:48:02.4441700Z + [[ inductor_torchbench_perf_cpu_x86_zen == *perf* ]]
2025-09-07T07:48:02.4442389Z + [[ inductor_torchbench_perf_cpu_x86_zen == *b200* ]]
2025-09-07T07:48:02.4443669Z + test_single_dynamo_benchmark dashboard torchbench 1
2025-09-07T07:48:02.4446586Z ++ pwd
2025-09-07T07:48:02.4449574Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2025-09-07T07:48:02.4449951Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2025-09-07T07:48:02.4471368Z + local name=dashboard
2025-09-07T07:48:02.4471576Z + shift
2025-09-07T07:48:02.4471738Z + local suite=torchbench
2025-09-07T07:48:02.4471979Z + shift
2025-09-07T07:48:02.4472119Z + local shard_id=1
2025-09-07T07:48:02.4472271Z + shift
2025-09-07T07:48:02.4472420Z + partition_flags=()
2025-09-07T07:48:02.4472594Z + local partition_flags
2025-09-07T07:48:02.4472767Z + [[ -n 4 ]]
2025-09-07T07:48:02.4472907Z + [[ -n 1 ]]
2025-09-07T07:48:02.4473174Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2025-09-07T07:48:02.4473542Z + [[ inductor_torchbench_perf_cpu_x86_zen == *perf_compare* ]]
2025-09-07T07:48:02.4473885Z + [[ inductor_torchbench_perf_cpu_x86_zen == *perf* ]]
2025-09-07T07:48:02.4474223Z + test_perf_for_dashboard torchbench --device cpu --total-partitions 4 --partition-id 1
2025-09-07T07:48:02.4476916Z ++ pwd
2025-09-07T07:48:02.4479734Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2025-09-07T07:48:02.4480021Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2025-09-07T07:48:02.4499104Z + local suite=torchbench
2025-09-07T07:48:02.4499267Z + shift
2025-09-07T07:48:02.4499407Z + local backend=inductor
2025-09-07T07:48:02.4499573Z + modes=()
2025-09-07T07:48:02.4499717Z + local modes
2025-09-07T07:48:02.4500127Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *training-true* ]]
2025-09-07T07:48:02.4500819Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *inference-true* ]]
2025-09-07T07:48:02.4501251Z + modes+=(inference)
2025-09-07T07:48:02.4501439Z + targets=('accuracy' 'performance')
2025-09-07T07:48:02.4501632Z + local targets
2025-09-07T07:48:02.4501779Z + local device=cuda
2025-09-07T07:48:02.4501964Z + [[ inductor_torchbench_perf_cpu_x86_zen == *cpu* ]]
2025-09-07T07:48:02.4502225Z + [[ inductor_torchbench_perf_cpu_x86_zen == *cpu_x86_zen* ]]
2025-09-07T07:48:02.4502446Z + device=cpu_x86_zen
2025-09-07T07:48:02.4502614Z + test_inductor_set_cpu_affinity
2025-09-07T07:48:02.4504533Z ++ find /usr/lib -name libjemalloc.so.2
2025-09-07T07:48:02.4860731Z + JEMALLOC_LIB=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
2025-09-07T07:48:02.4861039Z + export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2:
2025-09-07T07:48:02.4861312Z + LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2:
2025-09-07T07:48:02.4861718Z + export MALLOC_CONF=oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:-1,muzzy_decay_ms:-1
2025-09-07T07:48:02.4862239Z + MALLOC_CONF=oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:-1,muzzy_decay_ms:-1
2025-09-07T07:48:02.4863892Z + [[ inductor_torchbench_perf_cpu_x86_zen != *aarch64* ]]
2025-09-07T07:48:02.4869632Z +++ which python
2025-09-07T07:48:02.4894814Z ++ dirname /opt/conda/envs/py_3.9/bin/python
2025-09-07T07:48:02.4925780Z + IOMP_LIB=/opt/conda/envs/py_3.9/bin/../lib/libiomp5.so
2025-09-07T07:48:02.4926248Z + export LD_PRELOAD=/opt/conda/envs/py_3.9/bin/../lib/libiomp5.so:/usr/lib/x86_64-linux-gnu/libjemalloc.so.2:
2025-09-07T07:48:02.4926719Z + LD_PRELOAD=/opt/conda/envs/py_3.9/bin/../lib/libiomp5.so:/usr/lib/x86_64-linux-gnu/libjemalloc.so.2:
2025-09-07T07:48:02.4927144Z + export KMP_AFFINITY=granularity=fine,compact,1,0
2025-09-07T07:48:02.4927407Z + KMP_AFFINITY=granularity=fine,compact,1,0
2025-09-07T07:48:02.4927628Z + export KMP_BLOCKTIME=1
2025-09-07T07:48:02.4927795Z + KMP_BLOCKTIME=1
2025-09-07T07:48:02.4930980Z ++ nproc
2025-09-07T07:48:02.4964817Z + cpus=96
2025-09-07T07:48:02.4971936Z ++ lscpu
2025-09-07T07:48:02.4973633Z ++ grep 'Thread(s) per core:'
2025-09-07T07:48:02.4974851Z ++ awk '{print $4}'
2025-09-07T07:48:02.5581609Z + thread_per_core=1
2025-09-07T07:48:02.5581806Z + cores=96
2025-09-07T07:48:02.5582004Z + [[ inductor_torchbench_perf_cpu_x86_zen == *aarch64* ]]
2025-09-07T07:48:02.5582239Z + export OMP_NUM_THREADS=96
2025-09-07T07:48:02.5582413Z + OMP_NUM_THREADS=96
2025-09-07T07:48:02.5586272Z ++ python -c 'import os; print(min(os.sched_getaffinity(0)))'
2025-09-07T07:48:02.5844414Z + start_cpu=0
2025-09-07T07:48:02.5848816Z ++ python -c 'import os; print(max(os.sched_getaffinity(0)))'
2025-09-07T07:48:02.6112074Z + end_cpu=94
2025-09-07T07:48:02.6112262Z + export 'TASKSET=taskset -c 0-94'
2025-09-07T07:48:02.6112474Z + TASKSET='taskset -c 0-94'
2025-09-07T07:48:02.6112668Z + for mode in "${modes[@]}"
2025-09-07T07:48:02.6112834Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2025-09-07T07:48:02.6113038Z + [[ cpu_x86_zen == \c\p\u\_\x\8\6 ]]
2025-09-07T07:48:02.6113229Z + dtype=bfloat16
2025-09-07T07:48:02.6113387Z + for target in "${targets[@]}"
2025-09-07T07:48:02.6113588Z + target_flag=('--accuracy')
2025-09-07T07:48:02.6113766Z + local target_flag
2025-09-07T07:48:02.6113931Z + [[ accuracy == \p\e\r\f\o\r\m\a\n\c\e ]]
2025-09-07T07:48:02.6114128Z + [[ accuracy == \a\c\c\u\r\a\c\y ]]
2025-09-07T07:48:02.6114329Z + target_flag+=(--no-translation-validation)
2025-09-07T07:48:02.6114796Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *freezing-true* ]]
2025-09-07T07:48:02.6115477Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *default-true* ]]
2025-09-07T07:48:02.6116661Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv
2025-09-07T07:48:03.0112174Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:48:03.0113066Z   import pynvml  # type: ignore[import]
2025-09-07T07:48:05.5005237Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:48:05.5006087Z   import pynvml  # type: ignore[import]
2025-09-07T07:48:07.5021202Z 
2025-09-07T07:48:12.7974006Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:48:12.7974360Z loading model: 0it [00:05, ?it/s]
2025-09-07T07:48:12.8250561Z cpu  eval  dlrm                               
2025-09-07T07:48:12.9586907Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:48:12.9867915Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:48:13.0375168Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:48:28.5256210Z pass
2025-09-07T07:48:28.5256607Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:48:30.8274783Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:48:30.8275660Z   import pynvml  # type: ignore[import]
2025-09-07T07:48:32.8324119Z 
2025-09-07T07:48:34.1277714Z loading model: 0it [00:00, ?it/s]Downloading https://doctr-static.mindee.com/models?id=v0.7.0/db_resnet50-79bd7d70.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/db_resnet50-79bd7d70.pt
2025-09-07T07:48:34.3819083Z 
2025-09-07T07:48:34.3819092Z 
2025-09-07T07:48:34.4820760Z   0% 0/102021912 [00:00<?, ?it/s]
2025-09-07T07:48:34.4820969Z 
2025-09-07T07:48:34.5820683Z  31% 31491072/102021912 [00:00<00:00, 314859223.15it/s]
2025-09-07T07:48:34.5820875Z 
2025-09-07T07:48:34.6821385Z  64% 64956416/102021912 [00:00<00:00, 326501363.42it/s]
2025-09-07T07:48:34.6821560Z 
2025-09-07T07:48:34.6935696Z  96% 98355200/102021912 [00:00<00:00, 329918782.80it/s]
2025-09-07T07:48:34.6935963Z 102022144it [00:00, 327320177.15it/s]                  
2025-09-07T07:48:35.9469735Z Downloading https://doctr-static.mindee.com/models?id=v0.3.1/crnn_vgg16_bn-9762b0b0.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/crnn_vgg16_bn-9762b0b0.pt
2025-09-07T07:48:36.1859094Z 
2025-09-07T07:48:36.1859106Z 
2025-09-07T07:48:36.2861284Z   0% 0/63286381 [00:00<?, ?it/s]
2025-09-07T07:48:36.2861482Z 
2025-09-07T07:48:36.3798733Z  50% 31734784/63286381 [00:00<00:00, 317318637.30it/s]
2025-09-07T07:48:36.3799074Z 63287296it [00:00, 326487583.28it/s]                  
2025-09-07T07:48:36.7040919Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T07:48:37.3320487Z 
2025-09-07T07:48:37.3320997Z loading model: 0it [00:04, ?it/s]
2025-09-07T07:48:37.3321427Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T07:48:37.3321781Z cpu  eval  doctr_det_predictor                
2025-09-07T07:48:51.9075685Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:48:51.9076192Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T07:48:52.8845137Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:48:53.7825997Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:49:48.8118661Z pass
2025-09-07T07:49:48.8121717Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:49:51.7904742Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:49:51.7905609Z   import pynvml  # type: ignore[import]
2025-09-07T07:49:53.8127170Z 
2025-09-07T07:49:56.4630914Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T07:49:56.5054758Z 
2025-09-07T07:49:56.5054950Z loading model: 0it [00:02, ?it/s]
2025-09-07T07:49:56.5055328Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T07:49:56.5055695Z cpu  eval  doctr_reco_predictor               
2025-09-07T07:49:56.9865710Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:49:56.9866280Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T07:49:57.0473563Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:49:57.0949456Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:14.0927249Z pass
2025-09-07T07:50:14.0927644Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:16.3142858Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:50:16.3144365Z   import pynvml  # type: ignore[import]
2025-09-07T07:50:18.3167920Z 
2025-09-07T07:50:18.3716881Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T07:50:18.3717551Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T07:50:18.3718160Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T07:50:18.3718754Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T07:50:19.2816228Z 
2025-09-07T07:50:19.2816601Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:50:19.2830565Z cpu  eval  drq                                
2025-09-07T07:50:19.3010472Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:19.3117753Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:19.3193735Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:31.3589858Z pass
2025-09-07T07:50:31.3593175Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:33.3737646Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:50:33.3738534Z   import pynvml  # type: ignore[import]
2025-09-07T07:50:35.3836279Z 
2025-09-07T07:50:36.9525268Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:50:36.9525605Z loading model: 0it [00:01, ?it/s]
2025-09-07T07:50:36.9709592Z cpu  eval  fastNLP_Bert                       
2025-09-07T07:50:37.9697872Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:38.3699203Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:38.7680385Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:38.8558935Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T07:50:38.8559604Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T07:50:38.8560104Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T07:50:38.8560611Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T07:50:38.8561539Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T07:50:38.8562016Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T07:50:38.8562471Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T07:50:38.8563152Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T07:50:38.8563824Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(*inputs)
2025-09-07T07:50:38.8564477Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T07:50:38.8565291Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     sequence_output = self.bert(words)
2025-09-07T07:50:38.8566001Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T07:50:38.8566691Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.model(words)
2025-09-07T07:50:38.8567375Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T07:50:38.8569000Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T07:50:38.8569619Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T07:50:38.8569988Z W0907 07:50:38.855282 15279 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T07:50:55.8175763Z pass
2025-09-07T07:50:55.8179254Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:50:57.9961463Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:50:57.9962350Z   import pynvml  # type: ignore[import]
2025-09-07T07:50:59.9969660Z 
2025-09-07T07:51:00.4342969Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:51:00.4343383Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:51:00.4375311Z cpu  eval  functorch_dp_cifar10               
2025-09-07T07:51:00.5038558Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:00.5309444Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:00.5535366Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:22.1106748Z pass
2025-09-07T07:51:22.1109519Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:24.1219936Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:51:24.1220792Z   import pynvml  # type: ignore[import]
2025-09-07T07:51:26.1279986Z 
2025-09-07T07:51:26.3175181Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:51:26.3178552Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:51:26.3183544Z cpu  eval  functorch_maml_omniglot            
2025-09-07T07:51:26.3281441Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:26.3359209Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:26.3406801Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:35.3855579Z pass
2025-09-07T07:51:35.3857904Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:37.3249164Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:51:37.3250031Z   import pynvml  # type: ignore[import]
2025-09-07T07:51:39.3270519Z 
2025-09-07T07:51:40.0534564Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:51:40.0534763Z 
2025-09-07T07:51:40.0570960Z config.json:   0% 0.00/684 [00:00<?, ?B/s]
2025-09-07T07:51:40.0571205Z config.json: 100% 684/684 [00:00<00:00, 186kB/s]
2025-09-07T07:51:41.0855922Z 
2025-09-07T07:51:41.0856301Z loading model: 0it [00:01, ?it/s]
2025-09-07T07:51:41.0897072Z cpu  eval  hf_Albert                          
2025-09-07T07:51:42.0060378Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:42.2332627Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:42.4504263Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:56.8305979Z pass
2025-09-07T07:51:56.8306363Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:51:59.2007745Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:51:59.2008605Z   import pynvml  # type: ignore[import]
2025-09-07T07:52:01.2071460Z 
2025-09-07T07:52:01.8727182Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:52:01.8727389Z 
2025-09-07T07:52:01.8727742Z config.json: 0.00B [00:00, ?B/s]
2025-09-07T07:52:01.8727955Z config.json: 1.72kB [00:00, 16.5MB/s]
2025-09-07T07:52:05.4668365Z 
2025-09-07T07:52:05.4669038Z loading model: 0it [00:04, ?it/s]
2025-09-07T07:52:05.4994667Z cpu  eval  hf_Bart                            
2025-09-07T07:52:06.8473996Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:07.1983019Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:07.5448667Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:26.8125407Z pass
2025-09-07T07:52:26.8125808Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:29.4322847Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:52:29.4323741Z   import pynvml  # type: ignore[import]
2025-09-07T07:52:31.4366362Z 
2025-09-07T07:52:34.5762255Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:52:34.5762553Z loading model: 0it [00:03, ?it/s]
2025-09-07T07:52:34.5998837Z cpu  eval  hf_Bert                            
2025-09-07T07:52:35.6330626Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:35.9151121Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:36.1906535Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:49.4020270Z pass
2025-09-07T07:52:49.4020666Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:52:51.9035201Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:52:51.9036074Z   import pynvml  # type: ignore[import]
2025-09-07T07:52:53.9043947Z 
2025-09-07T07:52:59.2408719Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:52:59.2409052Z loading model: 0it [00:05, ?it/s]
2025-09-07T07:52:59.2804297Z cpu  eval  hf_Bert_large                      
2025-09-07T07:53:01.5450179Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:53:02.1944668Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:53:02.9034555Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:53:21.8587703Z pass
2025-09-07T07:53:21.8588090Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:53:24.5881784Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:53:24.5882654Z   import pynvml  # type: ignore[import]
2025-09-07T07:53:26.5944567Z 
2025-09-07T07:53:27.9952083Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T07:53:27.9953422Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T07:53:27.9954242Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T07:53:27.9954887Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T07:53:27.9956171Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T07:53:27.9957395Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T07:53:27.9958209Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T07:53:27.9958863Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T07:53:29.8226768Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T07:53:29.8227898Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T07:53:31.7080869Z 
2025-09-07T07:53:31.7081460Z loading model: 0it [00:05, ?it/s]
2025-09-07T07:53:31.7374641Z cpu  eval  hf_BigBird                         
2025-09-07T07:53:34.2347955Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:53:37.3109447Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:53:40.3743322Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:19.1345405Z W0907 07:54:19.133592 23158 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] torch._dynamo hit config.recompile_limit (8)
2025-09-07T07:54:19.1346922Z W0907 07:54:19.133592 23158 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    function: 'forward' (/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py:417)
2025-09-07T07:54:19.1348732Z W0907 07:54:19.133592 23158 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    last reason: 10/7: self.seed == 7                                           # context_layer, attention_probs = self.bigbird_block_sparse_attention(  # transformers/models/big_bird/modeling_big_bird.py:455 in forward (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
2025-09-07T07:54:19.1350410Z W0907 07:54:19.133592 23158 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-09-07T07:54:19.1351240Z W0907 07:54:19.133592 23158 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2025-09-07T07:54:23.7315074Z pass
2025-09-07T07:54:23.7315471Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:26.6235957Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:54:26.6236819Z   import pynvml  # type: ignore[import]
2025-09-07T07:54:28.6354705Z 
2025-09-07T07:54:29.3257631Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:54:29.3257831Z 
2025-09-07T07:54:29.3295239Z config.json:   0% 0.00/483 [00:00<?, ?B/s]
2025-09-07T07:54:29.3295484Z config.json: 100% 483/483 [00:00<00:00, 128kB/s]
2025-09-07T07:54:31.2766680Z 
2025-09-07T07:54:31.2767267Z loading model: 0it [00:02, ?it/s]
2025-09-07T07:54:31.2902143Z cpu  eval  hf_DistilBert                      
2025-09-07T07:54:31.8805496Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:32.0449600Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:32.1994124Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:42.3783706Z pass
2025-09-07T07:54:42.3784104Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:44.8333420Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:54:44.8334798Z   import pynvml  # type: ignore[import]
2025-09-07T07:54:46.8428291Z 
2025-09-07T07:54:47.5191614Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:54:47.5191819Z 
2025-09-07T07:54:47.5228905Z config.json:   0% 0.00/665 [00:00<?, ?B/s]
2025-09-07T07:54:47.5229243Z config.json: 100% 665/665 [00:00<00:00, 176kB/s]
2025-09-07T07:54:51.5342293Z 
2025-09-07T07:54:51.5342834Z loading model: 0it [00:04, ?it/s]
2025-09-07T07:54:51.5526023Z cpu  eval  hf_GPT2                            
2025-09-07T07:54:52.0882832Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:52.2575621Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:54:52.4116286Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:55:12.0453409Z pass
2025-09-07T07:55:12.0455978Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:55:14.6115433Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:55:14.6116426Z   import pynvml  # type: ignore[import]
2025-09-07T07:55:16.6200025Z 
2025-09-07T07:55:17.2916874Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:55:17.2917070Z 
2025-09-07T07:55:17.2952903Z config.json:   0% 0.00/666 [00:00<?, ?B/s]
2025-09-07T07:55:17.2953147Z config.json: 100% 666/666 [00:00<00:00, 180kB/s]
2025-09-07T07:55:30.9568673Z 
2025-09-07T07:55:30.9570667Z loading model: 0it [00:14, ?it/s]
2025-09-07T07:55:31.0186028Z cpu  eval  hf_GPT2_large                      
2025-09-07T07:55:31.4086531Z pass_due_to_skip
2025-09-07T07:55:31.4090900Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:55:33.8258306Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:55:33.8259165Z   import pynvml  # type: ignore[import]
2025-09-07T07:55:35.8303970Z 
2025-09-07T07:55:36.4931380Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:55:36.4931584Z 
2025-09-07T07:55:36.4967583Z config.json:   0% 0.00/694 [00:00<?, ?B/s]
2025-09-07T07:55:36.4967818Z config.json: 100% 694/694 [00:00<00:00, 188kB/s]
2025-09-07T07:55:39.8963001Z 
2025-09-07T07:55:39.8963555Z loading model: 0it [00:04, ?it/s]
2025-09-07T07:55:39.9267588Z cpu  eval  hf_Longformer                      
2025-09-07T07:55:41.5960088Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:55:42.4081048Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:55:43.1934322Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:55:43.3610326Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T07:55:43.3610977Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T07:55:43.3611485Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T07:55:43.3611971Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T07:55:43.3612541Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T07:55:43.3613037Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T07:55:43.3613861Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T07:55:43.3614540Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T07:55:43.3615218Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(**inputs)
2025-09-07T07:55:43.3615990Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T07:55:43.3616764Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.longformer(
2025-09-07T07:55:43.3617697Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T07:55:43.3618457Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     encoder_outputs = self.encoder(
2025-09-07T07:55:43.3619204Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1244, in forward
2025-09-07T07:55:43.3620002Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2025-09-07T07:55:43.3620493Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T07:55:43.3620867Z W0907 07:55:43.360399 27868 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T07:57:08.6582400Z pass
2025-09-07T07:57:08.6582831Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:57:11.8157551Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:57:11.8158416Z   import pynvml  # type: ignore[import]
2025-09-07T07:57:13.8144269Z 
2025-09-07T07:57:15.1887289Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T07:57:15.1888866Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T07:57:16.0264554Z 
2025-09-07T07:57:16.0264960Z loading model: 0it [00:02, ?it/s]
2025-09-07T07:57:16.0317164Z cpu  eval  hf_Reformer                        
2025-09-07T07:57:16.5553553Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:57:18.0350456Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:57:19.5095255Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:57:26.2076725Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T07:57:26.2077450Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T07:57:26.2078441Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] or:
2025-09-07T07:57:26.2078937Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T07:57:26.2079522Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] to include these operations in the captured graph.
2025-09-07T07:57:26.2079998Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T07:57:26.2080449Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break: from user code at:
2025-09-07T07:57:26.2081291Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2132, in torch_dynamo_resume_in_forward_at_2104
2025-09-07T07:57:26.2082320Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     embedding_output = self.embeddings(
2025-09-07T07:57:26.2083077Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 369, in forward
2025-09-07T07:57:26.2083870Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     position_embeddings = self.position_embeddings(position_ids)
2025-09-07T07:57:26.2084657Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T07:57:26.2085407Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     max_position_id = position_ids.max().item()
2025-09-07T07:57:26.2085880Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T07:57:26.2086245Z W0907 07:57:26.206971 29119 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T07:57:47.9366265Z pass
2025-09-07T07:57:47.9366663Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:57:50.2592879Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:57:50.2593754Z   import pynvml  # type: ignore[import]
2025-09-07T07:57:52.2730236Z 
2025-09-07T07:57:52.9486405Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:57:52.9486598Z 
2025-09-07T07:57:52.9524390Z config.json:   0% 0.00/615 [00:00<?, ?B/s]
2025-09-07T07:57:52.9524675Z config.json: 100% 615/615 [00:00<00:00, 160kB/s]
2025-09-07T07:57:58.0613250Z 
2025-09-07T07:57:58.0614997Z loading model: 0it [00:05, ?it/s]
2025-09-07T07:57:58.0901515Z cpu  eval  hf_Roberta_base                    
2025-09-07T07:57:59.9850998Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:00.5177371Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:01.0601546Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:16.4687486Z pass
2025-09-07T07:58:16.4689797Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:19.8907662Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:58:19.8911459Z   import pynvml  # type: ignore[import]
2025-09-07T07:58:21.9023344Z 
2025-09-07T07:58:22.5832799Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:58:22.5833013Z 
2025-09-07T07:58:22.5870521Z config.json:   0% 0.00/1.21k [00:00<?, ?B/s]
2025-09-07T07:58:22.5870782Z config.json: 100% 1.21k/1.21k [00:00<00:00, 316kB/s]
2025-09-07T07:58:24.7761081Z 
2025-09-07T07:58:24.7761531Z loading model: 0it [00:02, ?it/s]
2025-09-07T07:58:24.7966749Z cpu  eval  hf_T5                              
2025-09-07T07:58:25.8388464Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:26.2985798Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:26.7512529Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:54.3222825Z pass
2025-09-07T07:58:54.3223721Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:58:56.9315493Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T07:58:56.9316367Z   import pynvml  # type: ignore[import]
2025-09-07T07:58:58.9458822Z 
2025-09-07T07:58:59.6292666Z loading model: 0it [00:00, ?it/s]
2025-09-07T07:58:59.6292857Z 
2025-09-07T07:58:59.6330612Z config.json:   0% 0.00/1.21k [00:00<?, ?B/s]
2025-09-07T07:58:59.6330872Z config.json: 100% 1.21k/1.21k [00:00<00:00, 316kB/s]
2025-09-07T07:59:08.7082362Z 
2025-09-07T07:59:08.7083153Z loading model: 0it [00:09, ?it/s]
2025-09-07T07:59:08.7495090Z cpu  eval  hf_T5_base                         
2025-09-07T07:59:23.2949872Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:59:29.7510435Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T07:59:36.2398049Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:00:47.6829939Z pass
2025-09-07T08:00:47.6830340Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:00:53.8905599Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:00:53.8906477Z   import pynvml  # type: ignore[import]
2025-09-07T08:00:55.8925013Z 
2025-09-07T08:00:56.5722530Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:00:56.5722731Z 
2025-09-07T08:00:56.5760097Z config.json:   0% 0.00/1.21k [00:00<?, ?B/s]
2025-09-07T08:00:56.5760472Z config.json: 100% 1.21k/1.21k [00:00<00:00, 318kB/s]
2025-09-07T08:01:08.3217428Z 
2025-09-07T08:01:08.3218297Z loading model: 0it [00:12, ?it/s]
2025-09-07T08:01:08.4040352Z cpu  eval  hf_T5_large                        
2025-09-07T08:01:08.7988261Z pass_due_to_skip
2025-09-07T08:01:08.7991538Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:01:11.0966497Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:01:11.0967375Z   import pynvml  # type: ignore[import]
2025-09-07T08:01:13.1015125Z 
2025-09-07T08:01:13.7672954Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:01:13.7673155Z 
2025-09-07T08:01:13.7673975Z config.json: 0.00B [00:00, ?B/s]
2025-09-07T08:01:13.7674637Z config.json: 2.26kB [00:00, 17.5MB/s]
2025-09-07T08:01:19.3376841Z 
2025-09-07T08:01:19.3377303Z loading model: 0it [00:06, ?it/s]
2025-09-07T08:01:19.3753356Z cpu  eval  hf_distil_whisper                  
2025-09-07T08:01:26.5716405Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:01:28.6512930Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:01:30.7199103Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:02.7368120Z pass
2025-09-07T08:02:02.7368520Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:05.6811668Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:02:05.6812561Z   import pynvml  # type: ignore[import]
2025-09-07T08:02:07.6924097Z 
2025-09-07T08:02:07.8436326Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:02:07.8436609Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:02:07.8436948Z cpu  eval  lennard_jones                      
2025-09-07T08:02:07.8447855Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:07.8474764Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:07.8490262Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:14.4737365Z pass
2025-09-07T08:02:14.4740014Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:15.9739960Z accuracy            pass_rate=91.30%
2025-09-07T08:02:15.9743768Z calls_captured      gmean=0.00x mean=411.261x
2025-09-07T08:02:15.9746749Z unique_graphs       gmean=0.00x mean=2.957x
2025-09-07T08:02:15.9748944Z graph_breaks        gmean=0.00x mean=2.174x
2025-09-07T08:02:15.9751003Z unique_graph_breaks gmean=0.00x mean=0.565x
2025-09-07T08:02:15.9753086Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T08:02:15.9755228Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T08:02:15.9757277Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T08:02:15.9758209Z compilation_latency mean=23.003 seconds
2025-09-07T08:02:16.4953667Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *cudagraphs-true* ]]
2025-09-07T08:02:16.4954431Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *dynamic-true* ]]
2025-09-07T08:02:16.4955704Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --dynamic-shapes --dynamic-batch-only --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv
2025-09-07T08:02:16.8937529Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:02:16.8938459Z   import pynvml  # type: ignore[import]
2025-09-07T08:02:19.3301414Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:02:19.3302275Z   import pynvml  # type: ignore[import]
2025-09-07T08:02:21.3297198Z 
2025-09-07T08:02:26.5742390Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:02:26.5742724Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:02:26.6019075Z cpu  eval  dlrm                               
2025-09-07T08:02:26.7351302Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:26.7632357Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:26.7888574Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:29.0687203Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0687500Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0687742Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0687945Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0688141Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0688340Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0688526Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0689133Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0689369Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0689759Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0690088Z     return mod(*inputs)
2025-09-07T08:02:29.0690379Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0690726Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0691090Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 360, in sequential_forward
2025-09-07T08:02:29.0691430Z     x = self.apply_mlp(dense_x, self.bot_l)
2025-09-07T08:02:29.0691750Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0692053Z     return layers(x)
2025-09-07T08:02:29.0692150Z 
2025-09-07T08:02:29.0692256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0692627Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0692946Z     return mod(*inputs)
2025-09-07T08:02:29.0693221Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0693553Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0693901Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0694234Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0694548Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0694894Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0695057Z 
2025-09-07T08:02:29.0695159Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0695506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0695921Z     return mod(*inputs)
2025-09-07T08:02:29.0696200Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0696554Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0696929Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0697287Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0697612Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0698024Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0698215Z 
2025-09-07T08:02:29.0698486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0698824Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0699130Z     return mod(*inputs)
2025-09-07T08:02:29.0699569Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0699932Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0700273Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0700625Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0700942Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0701304Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0701487Z 
2025-09-07T08:02:29.0704326Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0704736Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0705053Z     return mod(*inputs)
2025-09-07T08:02:29.0705310Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0707629Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0707987Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0708318Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0708621Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0708958Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0709120Z 
2025-09-07T08:02:29.0709216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0709564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0709888Z     return mod(*inputs)
2025-09-07T08:02:29.0710168Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0710497Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0710839Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0711177Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0711487Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0711833Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0711991Z 
2025-09-07T08:02:29.0712099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0712450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0712759Z     return mod(*inputs)
2025-09-07T08:02:29.0713024Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0713344Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0713688Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0714013Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0714321Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0714660Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0714817Z 
2025-09-07T08:02:29.0714917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0715259Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0715566Z     return mod(*inputs)
2025-09-07T08:02:29.0715823Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0716140Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0716478Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0716807Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0717098Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0717551Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0717706Z 
2025-09-07T08:02:29.0717804Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0718140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0718444Z     return mod(*inputs)
2025-09-07T08:02:29.0718694Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0719012Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0719346Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T08:02:29.0719674Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T08:02:29.0719965Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T08:02:29.0720295Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T08:02:29.0720454Z 
2025-09-07T08:02:29.0720613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0720959Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0721265Z     return mod(*inputs)
2025-09-07T08:02:29.0721524Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0721842Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0722184Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 360, in sequential_forward
2025-09-07T08:02:29.0722520Z     x = self.apply_mlp(dense_x, self.bot_l)
2025-09-07T08:02:29.0722823Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0723117Z     return layers(x)
2025-09-07T08:02:29.0723212Z 
2025-09-07T08:02:29.0723311Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0723649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0723954Z     return mod(*inputs)
2025-09-07T08:02:29.0724250Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0724571Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0724910Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 360, in sequential_forward
2025-09-07T08:02:29.0725244Z     x = self.apply_mlp(dense_x, self.bot_l)
2025-09-07T08:02:29.0725543Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0725834Z     return layers(x)
2025-09-07T08:02:29.0725919Z 
2025-09-07T08:02:29.0725995Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0726195Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0726389Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0726582Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0726773Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0726969Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0727164Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0727360Z cudagraph partition due to non gpu ops
2025-09-07T08:02:29.0727568Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0727911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0728219Z     return mod(*inputs)
2025-09-07T08:02:29.0728473Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0728786Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0729115Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T08:02:29.0729448Z     z = self.interact_features(x, ly)
2025-09-07T08:02:29.0729764Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 318, in interact_features
2025-09-07T08:02:29.0730225Z     T = torch.cat([x] + ly, dim=1).view((batch_size, -1, d))
2025-09-07T08:02:29.0730381Z 
2025-09-07T08:02:29.0730477Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0730817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0731125Z     return mod(*inputs)
2025-09-07T08:02:29.0731380Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0731695Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0732023Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T08:02:29.0732344Z     z = self.interact_features(x, ly)
2025-09-07T08:02:29.0732655Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 320, in interact_features
2025-09-07T08:02:29.0732990Z     Z = torch.bmm(T, torch.transpose(T, 1, 2))
2025-09-07T08:02:29.0733132Z 
2025-09-07T08:02:29.0733294Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0733650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0733966Z     return mod(*inputs)
2025-09-07T08:02:29.0734226Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0734547Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0734883Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T08:02:29.0735214Z     z = self.interact_features(x, ly)
2025-09-07T08:02:29.0735526Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 337, in interact_features
2025-09-07T08:02:29.0735842Z     Zflat = Z[:, li, lj]
2025-09-07T08:02:29.0735941Z 
2025-09-07T08:02:29.0736032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0736376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0736694Z     return mod(*inputs)
2025-09-07T08:02:29.0736950Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0737265Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0737596Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T08:02:29.0737994Z     z = self.interact_features(x, ly)
2025-09-07T08:02:29.0738306Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 339, in interact_features
2025-09-07T08:02:29.0738630Z     R = torch.cat([x] + [Zflat], dim=1)
2025-09-07T08:02:29.0738751Z 
2025-09-07T08:02:29.0738843Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0739180Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0739505Z     return mod(*inputs)
2025-09-07T08:02:29.0739769Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0740090Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0740421Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0740742Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0741039Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0741332Z     return layers(x)
2025-09-07T08:02:29.0741420Z 
2025-09-07T08:02:29.0741512Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0741853Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0742167Z     return mod(*inputs)
2025-09-07T08:02:29.0742424Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0742739Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0743187Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0743514Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0743808Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0744096Z     return layers(x)
2025-09-07T08:02:29.0744185Z 
2025-09-07T08:02:29.0744278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0744618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0744927Z     return mod(*inputs)
2025-09-07T08:02:29.0745180Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0745496Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0745826Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0746147Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0746500Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0746817Z     return layers(x)
2025-09-07T08:02:29.0746911Z 
2025-09-07T08:02:29.0747008Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0747358Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0747678Z     return mod(*inputs)
2025-09-07T08:02:29.0747947Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0748257Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0748592Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0748911Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0749200Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0749492Z     return layers(x)
2025-09-07T08:02:29.0749580Z 
2025-09-07T08:02:29.0749673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0750011Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0750321Z     return mod(*inputs)
2025-09-07T08:02:29.0750579Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0750885Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0751219Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0751542Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0751829Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0752119Z     return layers(x)
2025-09-07T08:02:29.0752201Z 
2025-09-07T08:02:29.0752293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0752631Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0752939Z     return mod(*inputs)
2025-09-07T08:02:29.0753189Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0753497Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0753833Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0754151Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0754438Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0754727Z     return layers(x)
2025-09-07T08:02:29.0754813Z 
2025-09-07T08:02:29.0754903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0755239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0755542Z     return mod(*inputs)
2025-09-07T08:02:29.0755931Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0756239Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0756573Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0756892Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0757178Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0757469Z     return layers(x)
2025-09-07T08:02:29.0757551Z 
2025-09-07T08:02:29.0757640Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:02:29.0757971Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:02:29.0758279Z     return mod(*inputs)
2025-09-07T08:02:29.0758534Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T08:02:29.0758839Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T08:02:29.0759234Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T08:02:29.0759559Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T08:02:29.0759843Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T08:02:29.0760131Z     return layers(x)
2025-09-07T08:02:29.0760215Z 
2025-09-07T08:02:34.1738613Z pass
2025-09-07T08:02:34.1740990Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:36.3369792Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:02:36.3370872Z   import pynvml  # type: ignore[import]
2025-09-07T08:02:38.3349037Z 
2025-09-07T08:02:41.0502370Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:02:41.6766619Z 
2025-09-07T08:02:41.6767058Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:02:41.6767456Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:02:41.6767824Z cpu  eval  doctr_det_predictor                
2025-09-07T08:02:56.2751655Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:56.2752156Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:02:57.1822856Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:02:58.0871934Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:03:06.8346537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8347158Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8347634Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8347983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8348323Z     x = module(x)
2025-09-07T08:03:06.8348423Z 
2025-09-07T08:03:06.8348530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8349021Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8349489Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8349829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8350163Z     x = module(x)
2025-09-07T08:03:06.8350257Z 
2025-09-07T08:03:06.8350728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8351204Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8351647Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8351972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8352299Z     x = module(x)
2025-09-07T08:03:06.8352600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8352942Z     out = self.conv1(x)
2025-09-07T08:03:06.8353042Z 
2025-09-07T08:03:06.8353137Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8353599Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8354034Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8354525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8354868Z     x = module(x)
2025-09-07T08:03:06.8355167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8355513Z     out = self.conv2(out)
2025-09-07T08:03:06.8355627Z 
2025-09-07T08:03:06.8355731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8356204Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8356647Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8356977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8357314Z     x = module(x)
2025-09-07T08:03:06.8357617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8357965Z     out = self.conv3(out)
2025-09-07T08:03:06.8358066Z 
2025-09-07T08:03:06.8358162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8358634Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8359076Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8359403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8359727Z     x = module(x)
2025-09-07T08:03:06.8360019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8360365Z     identity = self.downsample(x)
2025-09-07T08:03:06.8360495Z 
2025-09-07T08:03:06.8360595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8361057Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8361502Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8361816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8362139Z     x = module(x)
2025-09-07T08:03:06.8362432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8362771Z     out = self.conv1(x)
2025-09-07T08:03:06.8362869Z 
2025-09-07T08:03:06.8362968Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8363435Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8363879Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8364211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8364619Z     x = module(x)
2025-09-07T08:03:06.8364921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8365303Z     out = self.conv2(out)
2025-09-07T08:03:06.8365412Z 
2025-09-07T08:03:06.8365505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8365981Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8366438Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8366888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8367230Z     x = module(x)
2025-09-07T08:03:06.8367532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8367875Z     out = self.conv3(out)
2025-09-07T08:03:06.8367973Z 
2025-09-07T08:03:06.8368152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8368620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8369057Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8369382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8369710Z     x = module(x)
2025-09-07T08:03:06.8369993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8370336Z     out = self.conv1(x)
2025-09-07T08:03:06.8370442Z 
2025-09-07T08:03:06.8370536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8370991Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8371440Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8371765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8372106Z     x = module(x)
2025-09-07T08:03:06.8372408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8372742Z     out = self.conv2(out)
2025-09-07T08:03:06.8372840Z 
2025-09-07T08:03:06.8372938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8373394Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8373828Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8374149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8374479Z     x = module(x)
2025-09-07T08:03:06.8374761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8375097Z     out = self.conv3(out)
2025-09-07T08:03:06.8375207Z 
2025-09-07T08:03:06.8375301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8375753Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8376191Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8376503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8376828Z     x = module(x)
2025-09-07T08:03:06.8377122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8377453Z     out = self.conv1(x)
2025-09-07T08:03:06.8377545Z 
2025-09-07T08:03:06.8377636Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8378161Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8379702Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8380027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8380357Z     x = module(x)
2025-09-07T08:03:06.8380642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8380988Z     out = self.conv2(out)
2025-09-07T08:03:06.8381096Z 
2025-09-07T08:03:06.8381190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8381644Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8382075Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8382383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8382712Z     x = module(x)
2025-09-07T08:03:06.8383070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8383408Z     out = self.conv3(out)
2025-09-07T08:03:06.8383503Z 
2025-09-07T08:03:06.8383594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8384046Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8384479Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8384794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8385116Z     x = module(x)
2025-09-07T08:03:06.8385398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8385744Z     identity = self.downsample(x)
2025-09-07T08:03:06.8385871Z 
2025-09-07T08:03:06.8385968Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8386422Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8386850Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8387161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8387485Z     x = module(x)
2025-09-07T08:03:06.8387775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8388107Z     out = self.conv1(x)
2025-09-07T08:03:06.8388196Z 
2025-09-07T08:03:06.8388290Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8388742Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8389172Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8389496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8389819Z     x = module(x)
2025-09-07T08:03:06.8390102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8390437Z     out = self.conv2(out)
2025-09-07T08:03:06.8390545Z 
2025-09-07T08:03:06.8390642Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8391101Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8391536Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8391855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8392184Z     x = module(x)
2025-09-07T08:03:06.8392481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8392895Z     out = self.conv3(out)
2025-09-07T08:03:06.8392992Z 
2025-09-07T08:03:06.8393083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8393542Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8393975Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8394295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8394615Z     x = module(x)
2025-09-07T08:03:06.8394897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8395231Z     out = self.conv1(x)
2025-09-07T08:03:06.8395323Z 
2025-09-07T08:03:06.8395420Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8395958Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8396387Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8396702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8397027Z     x = module(x)
2025-09-07T08:03:06.8397315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8397647Z     out = self.conv2(out)
2025-09-07T08:03:06.8397741Z 
2025-09-07T08:03:06.8397831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8398283Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8398716Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8399037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8399368Z     x = module(x)
2025-09-07T08:03:06.8399651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8399983Z     out = self.conv3(out)
2025-09-07T08:03:06.8400078Z 
2025-09-07T08:03:06.8400175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8400622Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8401045Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8401377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8401704Z     x = module(x)
2025-09-07T08:03:06.8401995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8402327Z     out = self.conv1(x)
2025-09-07T08:03:06.8402416Z 
2025-09-07T08:03:06.8402513Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8402969Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8403560Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8403881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8404202Z     x = module(x)
2025-09-07T08:03:06.8404490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8404819Z     out = self.conv2(out)
2025-09-07T08:03:06.8404914Z 
2025-09-07T08:03:06.8405010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8405463Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8405884Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8406318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8406645Z     x = module(x)
2025-09-07T08:03:06.8406937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8407267Z     out = self.conv3(out)
2025-09-07T08:03:06.8407363Z 
2025-09-07T08:03:06.8407453Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8407904Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8408332Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8408650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8408969Z     x = module(x)
2025-09-07T08:03:06.8409258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8409687Z     out = self.conv1(x)
2025-09-07T08:03:06.8409780Z 
2025-09-07T08:03:06.8409879Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8410335Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8410755Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8411071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8411392Z     x = module(x)
2025-09-07T08:03:06.8411682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8412008Z     out = self.conv2(out)
2025-09-07T08:03:06.8412108Z 
2025-09-07T08:03:06.8412197Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8412650Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8413082Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8413404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8413721Z     x = module(x)
2025-09-07T08:03:06.8414009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8414340Z     out = self.conv3(out)
2025-09-07T08:03:06.8414435Z 
2025-09-07T08:03:06.8414533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8414984Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8415410Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8415723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8416046Z     x = module(x)
2025-09-07T08:03:06.8416335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8416668Z     identity = self.downsample(x)
2025-09-07T08:03:06.8416789Z 
2025-09-07T08:03:06.8416877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8417333Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8417822Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8418141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8418459Z     x = module(x)
2025-09-07T08:03:06.8418749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8419083Z     out = self.conv1(x)
2025-09-07T08:03:06.8419174Z 
2025-09-07T08:03:06.8419354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8419805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8420227Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8420547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8420867Z     x = module(x)
2025-09-07T08:03:06.8421156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8421484Z     out = self.conv2(out)
2025-09-07T08:03:06.8421585Z 
2025-09-07T08:03:06.8421675Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8422125Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8422553Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8422940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8423264Z     x = module(x)
2025-09-07T08:03:06.8423559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8423888Z     out = self.conv3(out)
2025-09-07T08:03:06.8423983Z 
2025-09-07T08:03:06.8424079Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8424526Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8424953Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8425270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8425598Z     x = module(x)
2025-09-07T08:03:06.8425888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8426218Z     out = self.conv1(x)
2025-09-07T08:03:06.8426320Z 
2025-09-07T08:03:06.8426411Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8426860Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8427289Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8427605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8427928Z     x = module(x)
2025-09-07T08:03:06.8428228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8428564Z     out = self.conv2(out)
2025-09-07T08:03:06.8428659Z 
2025-09-07T08:03:06.8428758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8429211Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8429646Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8429971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8430299Z     x = module(x)
2025-09-07T08:03:06.8430590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8430917Z     out = self.conv3(out)
2025-09-07T08:03:06.8431021Z 
2025-09-07T08:03:06.8431114Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8431565Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8431999Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8432320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8432720Z     x = module(x)
2025-09-07T08:03:06.8433015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8433346Z     out = self.conv1(x)
2025-09-07T08:03:06.8433438Z 
2025-09-07T08:03:06.8433538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8433982Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8434412Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8434730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8435058Z     x = module(x)
2025-09-07T08:03:06.8435352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8435676Z     out = self.conv2(out)
2025-09-07T08:03:06.8435778Z 
2025-09-07T08:03:06.8435937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8436394Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8436824Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8437137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8437460Z     x = module(x)
2025-09-07T08:03:06.8437751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8438081Z     out = self.conv3(out)
2025-09-07T08:03:06.8438176Z 
2025-09-07T08:03:06.8438271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8438719Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8439145Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8439468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8439797Z     x = module(x)
2025-09-07T08:03:06.8440098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8440434Z     out = self.conv1(x)
2025-09-07T08:03:06.8440537Z 
2025-09-07T08:03:06.8440634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8441096Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8441534Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8441862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8442194Z     x = module(x)
2025-09-07T08:03:06.8442493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8442837Z     out = self.conv2(out)
2025-09-07T08:03:06.8442940Z 
2025-09-07T08:03:06.8443043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8443496Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8443936Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8444263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8444587Z     x = module(x)
2025-09-07T08:03:06.8444877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8445216Z     out = self.conv3(out)
2025-09-07T08:03:06.8445320Z 
2025-09-07T08:03:06.8445413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8445879Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8446403Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8446721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8447046Z     x = module(x)
2025-09-07T08:03:06.8447336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8447666Z     out = self.conv1(x)
2025-09-07T08:03:06.8447761Z 
2025-09-07T08:03:06.8447860Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8448307Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8448738Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8449058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8449450Z     x = module(x)
2025-09-07T08:03:06.8449741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8450072Z     out = self.conv2(out)
2025-09-07T08:03:06.8450180Z 
2025-09-07T08:03:06.8450273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8450731Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8451158Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8451473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8451796Z     x = module(x)
2025-09-07T08:03:06.8452083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8452419Z     out = self.conv3(out)
2025-09-07T08:03:06.8452516Z 
2025-09-07T08:03:06.8452618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8453069Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8453499Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8453824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8454154Z     x = module(x)
2025-09-07T08:03:06.8454434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8454770Z     out = self.conv1(x)
2025-09-07T08:03:06.8454866Z 
2025-09-07T08:03:06.8454956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8455405Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8455834Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8456148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8456469Z     x = module(x)
2025-09-07T08:03:06.8456757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8457092Z     out = self.conv2(out)
2025-09-07T08:03:06.8457186Z 
2025-09-07T08:03:06.8457276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8457777Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8458210Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8458534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8458858Z     x = module(x)
2025-09-07T08:03:06.8459143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8459559Z     out = self.conv3(out)
2025-09-07T08:03:06.8459663Z 
2025-09-07T08:03:06.8459755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8460207Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8460643Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8460957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8461281Z     x = module(x)
2025-09-07T08:03:06.8461577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8461918Z     identity = self.downsample(x)
2025-09-07T08:03:06.8462037Z 
2025-09-07T08:03:06.8462129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8462643Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8463085Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8463400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8463735Z     x = module(x)
2025-09-07T08:03:06.8464013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8464344Z     out = self.conv1(x)
2025-09-07T08:03:06.8464441Z 
2025-09-07T08:03:06.8464531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8464981Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8465409Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8465718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8466049Z     x = module(x)
2025-09-07T08:03:06.8466338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8466667Z     out = self.conv2(out)
2025-09-07T08:03:06.8466763Z 
2025-09-07T08:03:06.8466854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8467308Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8467735Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8468054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8468375Z     x = module(x)
2025-09-07T08:03:06.8468658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8468986Z     out = self.conv3(out)
2025-09-07T08:03:06.8469086Z 
2025-09-07T08:03:06.8469185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8469635Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8470057Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8481570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8481940Z     x = module(x)
2025-09-07T08:03:06.8482269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8482613Z     out = self.conv1(x)
2025-09-07T08:03:06.8482724Z 
2025-09-07T08:03:06.8482830Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8483327Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8483776Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8484227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8484559Z     x = module(x)
2025-09-07T08:03:06.8484876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8485224Z     out = self.conv2(out)
2025-09-07T08:03:06.8485331Z 
2025-09-07T08:03:06.8485443Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8485910Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8486344Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8486676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8487007Z     x = module(x)
2025-09-07T08:03:06.8487389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8487728Z     out = self.conv3(out)
2025-09-07T08:03:06.8487839Z 
2025-09-07T08:03:06.8487934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8488398Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8488838Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8489256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8489774Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8490309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8490818Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8491011Z 
2025-09-07T08:03:06.8491121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8491582Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8492010Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8492425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8492916Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8493414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8493913Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8494100Z 
2025-09-07T08:03:06.8494194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8494669Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8495123Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8495544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8496075Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8496592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8497097Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8497293Z 
2025-09-07T08:03:06.8497392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8497946Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8498466Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8498879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8499374Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8499868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8500364Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8500548Z 
2025-09-07T08:03:06.8500647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8501091Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8501524Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8501996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8502537Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8503207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8503711Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8503900Z 
2025-09-07T08:03:06.8504003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8504477Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8504918Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8505351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8505830Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8506318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8506803Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8506979Z 
2025-09-07T08:03:06.8507074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8507533Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8507959Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8508377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8508857Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8509334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8509818Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8509987Z 
2025-09-07T08:03:06.8510084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8510532Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8510961Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8511387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8511868Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8512347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8512958Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8513135Z 
2025-09-07T08:03:06.8513228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8513684Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8514118Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8514254Z 
2025-09-07T08:03:06.8514342Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8514795Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8515225Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8515354Z 
2025-09-07T08:03:06.8515458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8516033Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8516481Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8516605Z 
2025-09-07T08:03:06.8516704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8517157Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8517580Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8517708Z 
2025-09-07T08:03:06.8517797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8518247Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8518681Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8519020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8519355Z     x = module(x)
2025-09-07T08:03:06.8519449Z 
2025-09-07T08:03:06.8519539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8519993Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8520422Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8520738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8521061Z     x = module(x)
2025-09-07T08:03:06.8521147Z 
2025-09-07T08:03:06.8521236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8521688Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8522119Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8522435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8522759Z     x = module(x)
2025-09-07T08:03:06.8522843Z 
2025-09-07T08:03:06.8522931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8523382Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8523810Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8524117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8524440Z     x = module(x)
2025-09-07T08:03:06.8524738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8525072Z     out = self.conv1(x)
2025-09-07T08:03:06.8525167Z 
2025-09-07T08:03:06.8525256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8525785Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8526211Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8526530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8526854Z     x = module(x)
2025-09-07T08:03:06.8527139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8527469Z     out = self.relu(out)
2025-09-07T08:03:06.8527578Z 
2025-09-07T08:03:06.8527672Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8528127Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8528545Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8528933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8529264Z     x = module(x)
2025-09-07T08:03:06.8529566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8529900Z     out = self.conv2(out)
2025-09-07T08:03:06.8530004Z 
2025-09-07T08:03:06.8530096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8530550Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8530977Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8531302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8531625Z     x = module(x)
2025-09-07T08:03:06.8531910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8532259Z     out = self.relu(out)
2025-09-07T08:03:06.8532362Z 
2025-09-07T08:03:06.8532463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8532929Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8533356Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8533679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8534005Z     x = module(x)
2025-09-07T08:03:06.8534349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8534687Z     out = self.conv3(out)
2025-09-07T08:03:06.8534784Z 
2025-09-07T08:03:06.8534877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8535337Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8535769Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8536087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8536409Z     x = module(x)
2025-09-07T08:03:06.8536691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8537033Z     identity = self.downsample(x)
2025-09-07T08:03:06.8537152Z 
2025-09-07T08:03:06.8537252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8537752Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8538172Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8538492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8538808Z     x = module(x)
2025-09-07T08:03:06.8539172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8539494Z     out = self.conv1(x)
2025-09-07T08:03:06.8539586Z 
2025-09-07T08:03:06.8539676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8540124Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8540548Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8540856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8541168Z     x = module(x)
2025-09-07T08:03:06.8541442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8541771Z     out = self.relu(out)
2025-09-07T08:03:06.8541863Z 
2025-09-07T08:03:06.8541954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8542472Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8542907Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8543230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8543546Z     x = module(x)
2025-09-07T08:03:06.8543836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8544167Z     out = self.conv2(out)
2025-09-07T08:03:06.8544263Z 
2025-09-07T08:03:06.8544354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8544797Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8545215Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8545531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8545840Z     x = module(x)
2025-09-07T08:03:06.8546121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8546441Z     out = self.relu(out)
2025-09-07T08:03:06.8546536Z 
2025-09-07T08:03:06.8546628Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8547070Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8547484Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8547792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8548103Z     x = module(x)
2025-09-07T08:03:06.8548384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8548710Z     out = self.conv3(out)
2025-09-07T08:03:06.8548806Z 
2025-09-07T08:03:06.8548893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8549332Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8549748Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8550052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8550361Z     x = module(x)
2025-09-07T08:03:06.8550637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8550956Z     out = self.relu(out)
2025-09-07T08:03:06.8551047Z 
2025-09-07T08:03:06.8551136Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8551577Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8552078Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8552408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8552730Z     x = module(x)
2025-09-07T08:03:06.8553024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8553347Z     out = self.conv1(x)
2025-09-07T08:03:06.8553447Z 
2025-09-07T08:03:06.8553543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8553997Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8554421Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8554736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8555046Z     x = module(x)
2025-09-07T08:03:06.8555390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8555720Z     out = self.relu(out)
2025-09-07T08:03:06.8555817Z 
2025-09-07T08:03:06.8555910Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8556357Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8556774Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8557083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8557397Z     x = module(x)
2025-09-07T08:03:06.8557676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8558002Z     out = self.conv2(out)
2025-09-07T08:03:06.8558104Z 
2025-09-07T08:03:06.8558190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8558638Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8559055Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8559362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8559670Z     x = module(x)
2025-09-07T08:03:06.8559946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8560269Z     out = self.relu(out)
2025-09-07T08:03:06.8560361Z 
2025-09-07T08:03:06.8560449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8560889Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8561301Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8561618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8561932Z     x = module(x)
2025-09-07T08:03:06.8562208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8562527Z     out = self.conv3(out)
2025-09-07T08:03:06.8562626Z 
2025-09-07T08:03:06.8562712Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8563154Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8563576Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8563883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8564194Z     x = module(x)
2025-09-07T08:03:06.8564474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8565338Z     out = self.relu(out)
2025-09-07T08:03:06.8565431Z 
2025-09-07T08:03:06.8565521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8565965Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8566387Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8566699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8567014Z     x = module(x)
2025-09-07T08:03:06.8567300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8567616Z     out = self.conv1(x)
2025-09-07T08:03:06.8567714Z 
2025-09-07T08:03:06.8567802Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8568242Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8568733Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8569047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8569360Z     x = module(x)
2025-09-07T08:03:06.8569642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8569968Z     out = self.relu(out)
2025-09-07T08:03:06.8570058Z 
2025-09-07T08:03:06.8570150Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8570591Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8571007Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8571317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8571635Z     x = module(x)
2025-09-07T08:03:06.8571926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8572243Z     out = self.conv2(out)
2025-09-07T08:03:06.8572342Z 
2025-09-07T08:03:06.8572428Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8572870Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8573299Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8573601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8573912Z     x = module(x)
2025-09-07T08:03:06.8574202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8574585Z     out = self.relu(out)
2025-09-07T08:03:06.8574675Z 
2025-09-07T08:03:06.8574768Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8575211Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8575629Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8575938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8576250Z     x = module(x)
2025-09-07T08:03:06.8576532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8576857Z     out = self.conv3(out)
2025-09-07T08:03:06.8576957Z 
2025-09-07T08:03:06.8577042Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8577491Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8577952Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8578334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8578657Z     x = module(x)
2025-09-07T08:03:06.8578944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8579280Z     identity = self.downsample(x)
2025-09-07T08:03:06.8579401Z 
2025-09-07T08:03:06.8579495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8579939Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8580363Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8580673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8580991Z     x = module(x)
2025-09-07T08:03:06.8581269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8581671Z     out = self.conv1(x)
2025-09-07T08:03:06.8581770Z 
2025-09-07T08:03:06.8581859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8582309Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8582728Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8583034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8583350Z     x = module(x)
2025-09-07T08:03:06.8583634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8583957Z     out = self.relu(out)
2025-09-07T08:03:06.8584049Z 
2025-09-07T08:03:06.8584140Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8584582Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8585008Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8585321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8585633Z     x = module(x)
2025-09-07T08:03:06.8585910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8586233Z     out = self.conv2(out)
2025-09-07T08:03:06.8586327Z 
2025-09-07T08:03:06.8586412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8586854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8587278Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8587581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8587904Z     x = module(x)
2025-09-07T08:03:06.8588183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8588506Z     out = self.relu(out)
2025-09-07T08:03:06.8588594Z 
2025-09-07T08:03:06.8588686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8589131Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8589549Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8589852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8590162Z     x = module(x)
2025-09-07T08:03:06.8590436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8590755Z     out = self.conv3(out)
2025-09-07T08:03:06.8590847Z 
2025-09-07T08:03:06.8590931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8591514Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8591930Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8592237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8592548Z     x = module(x)
2025-09-07T08:03:06.8592828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8593147Z     out = self.relu(out)
2025-09-07T08:03:06.8593241Z 
2025-09-07T08:03:06.8593329Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8593770Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8594185Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8594554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8594869Z     x = module(x)
2025-09-07T08:03:06.8595144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8595461Z     out = self.conv1(x)
2025-09-07T08:03:06.8595561Z 
2025-09-07T08:03:06.8595658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8596099Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8596523Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8596834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8597160Z     x = module(x)
2025-09-07T08:03:06.8597438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8597774Z     out = self.relu(out)
2025-09-07T08:03:06.8597874Z 
2025-09-07T08:03:06.8597963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8598414Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8598843Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8599154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8599475Z     x = module(x)
2025-09-07T08:03:06.8599759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8600089Z     out = self.conv2(out)
2025-09-07T08:03:06.8600184Z 
2025-09-07T08:03:06.8600276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8600721Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8601145Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8601459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8601775Z     x = module(x)
2025-09-07T08:03:06.8602056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8602378Z     out = self.relu(out)
2025-09-07T08:03:06.8602475Z 
2025-09-07T08:03:06.8602562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8603005Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8603572Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8603891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8604323Z     x = module(x)
2025-09-07T08:03:06.8604620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8604954Z     out = self.conv3(out)
2025-09-07T08:03:06.8605055Z 
2025-09-07T08:03:06.8605157Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8605614Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8606078Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8606409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8606743Z     x = module(x)
2025-09-07T08:03:06.8607037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8607368Z     out = self.relu(out)
2025-09-07T08:03:06.8607473Z 
2025-09-07T08:03:06.8607568Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8608118Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8608555Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8608873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8609208Z     x = module(x)
2025-09-07T08:03:06.8609503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8609834Z     out = self.conv1(x)
2025-09-07T08:03:06.8609924Z 
2025-09-07T08:03:06.8610015Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8610468Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8610893Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8611214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8611530Z     x = module(x)
2025-09-07T08:03:06.8611810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8612135Z     out = self.relu(out)
2025-09-07T08:03:06.8612227Z 
2025-09-07T08:03:06.8612315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8612771Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8613200Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8613512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8613833Z     x = module(x)
2025-09-07T08:03:06.8614122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8614459Z     out = self.conv2(out)
2025-09-07T08:03:06.8614558Z 
2025-09-07T08:03:06.8614646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8615099Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8615529Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8615841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8616190Z     x = module(x)
2025-09-07T08:03:06.8616472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8616801Z     out = self.relu(out)
2025-09-07T08:03:06.8616900Z 
2025-09-07T08:03:06.8616991Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8617444Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8617989Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8618301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8618624Z     x = module(x)
2025-09-07T08:03:06.8618917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8619247Z     out = self.conv3(out)
2025-09-07T08:03:06.8619343Z 
2025-09-07T08:03:06.8619434Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8619885Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8620304Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8620619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8620941Z     x = module(x)
2025-09-07T08:03:06.8621288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8621626Z     out = self.relu(out)
2025-09-07T08:03:06.8621718Z 
2025-09-07T08:03:06.8621813Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8622264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8622688Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8623004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8623321Z     x = module(x)
2025-09-07T08:03:06.8623608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8623938Z     out = self.conv1(x)
2025-09-07T08:03:06.8624028Z 
2025-09-07T08:03:06.8624119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8624571Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8624999Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8625316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8625639Z     x = module(x)
2025-09-07T08:03:06.8625919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8626248Z     out = self.relu(out)
2025-09-07T08:03:06.8626341Z 
2025-09-07T08:03:06.8626435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8626881Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8627300Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8627620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8627940Z     x = module(x)
2025-09-07T08:03:06.8628231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8628567Z     out = self.conv2(out)
2025-09-07T08:03:06.8628662Z 
2025-09-07T08:03:06.8628753Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8629204Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8629625Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8629943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8630261Z     x = module(x)
2025-09-07T08:03:06.8630548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8630964Z     out = self.relu(out)
2025-09-07T08:03:06.8631070Z 
2025-09-07T08:03:06.8631168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8631616Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8632036Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8632353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8632672Z     x = module(x)
2025-09-07T08:03:06.8632963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8633291Z     out = self.conv3(out)
2025-09-07T08:03:06.8633386Z 
2025-09-07T08:03:06.8633477Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8633987Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8634427Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8634745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8635137Z     x = module(x)
2025-09-07T08:03:06.8635437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8635776Z     identity = self.downsample(x)
2025-09-07T08:03:06.8635896Z 
2025-09-07T08:03:06.8635995Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8636457Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8636884Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8637222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8637552Z     x = module(x)
2025-09-07T08:03:06.8637854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8638187Z     out = self.conv1(x)
2025-09-07T08:03:06.8638284Z 
2025-09-07T08:03:06.8638381Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8638840Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8639270Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8639593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8639911Z     x = module(x)
2025-09-07T08:03:06.8640204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8640537Z     out = self.relu(out)
2025-09-07T08:03:06.8640637Z 
2025-09-07T08:03:06.8640735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8641188Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8641610Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8641923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8642241Z     x = module(x)
2025-09-07T08:03:06.8642530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8642853Z     out = self.conv2(out)
2025-09-07T08:03:06.8642959Z 
2025-09-07T08:03:06.8643048Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8643495Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8643931Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8644335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8644651Z     x = module(x)
2025-09-07T08:03:06.8644945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8645273Z     out = self.relu(out)
2025-09-07T08:03:06.8645371Z 
2025-09-07T08:03:06.8645469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8645921Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8646343Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8646661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8646983Z     x = module(x)
2025-09-07T08:03:06.8647269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8647659Z     out = self.conv3(out)
2025-09-07T08:03:06.8647763Z 
2025-09-07T08:03:06.8647854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8648314Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8648743Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8649066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8649382Z     x = module(x)
2025-09-07T08:03:06.8649669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8650026Z     out = self.relu(out)
2025-09-07T08:03:06.8650120Z 
2025-09-07T08:03:06.8650216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8650673Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8651095Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8651412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8651732Z     x = module(x)
2025-09-07T08:03:06.8652019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8652344Z     out = self.conv1(x)
2025-09-07T08:03:06.8652443Z 
2025-09-07T08:03:06.8652533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8652988Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8653426Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8653742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8654062Z     x = module(x)
2025-09-07T08:03:06.8654352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8654681Z     out = self.relu(out)
2025-09-07T08:03:06.8654775Z 
2025-09-07T08:03:06.8654871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8655315Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8655743Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8656061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8656383Z     x = module(x)
2025-09-07T08:03:06.8656671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8656996Z     out = self.conv2(out)
2025-09-07T08:03:06.8657099Z 
2025-09-07T08:03:06.8657195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8657799Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8658231Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8658552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8658871Z     x = module(x)
2025-09-07T08:03:06.8659163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8659493Z     out = self.relu(out)
2025-09-07T08:03:06.8659586Z 
2025-09-07T08:03:06.8659682Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8660127Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8660560Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8660947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8661278Z     x = module(x)
2025-09-07T08:03:06.8661568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8661895Z     out = self.conv3(out)
2025-09-07T08:03:06.8661998Z 
2025-09-07T08:03:06.8662088Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8662536Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8662964Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8663285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8663602Z     x = module(x)
2025-09-07T08:03:06.8663892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8664233Z     out = self.relu(out)
2025-09-07T08:03:06.8664325Z 
2025-09-07T08:03:06.8664422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8664868Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8665296Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8665621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8665950Z     x = module(x)
2025-09-07T08:03:06.8666239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8666570Z     out = self.conv1(x)
2025-09-07T08:03:06.8666668Z 
2025-09-07T08:03:06.8666760Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8667210Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8667653Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8667977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8668308Z     x = module(x)
2025-09-07T08:03:06.8668603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8668942Z     out = self.relu(out)
2025-09-07T08:03:06.8669041Z 
2025-09-07T08:03:06.8669142Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8669593Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8670024Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8670344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8670748Z     x = module(x)
2025-09-07T08:03:06.8671036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8671374Z     out = self.conv2(out)
2025-09-07T08:03:06.8671483Z 
2025-09-07T08:03:06.8671574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8672031Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8672463Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8672776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8673103Z     x = module(x)
2025-09-07T08:03:06.8673395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8673729Z     out = self.relu(out)
2025-09-07T08:03:06.8673821Z 
2025-09-07T08:03:06.8673919Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8674440Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8674882Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8675208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8675532Z     x = module(x)
2025-09-07T08:03:06.8675819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8676154Z     out = self.conv3(out)
2025-09-07T08:03:06.8676257Z 
2025-09-07T08:03:06.8676345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8676797Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8677226Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8677540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8677867Z     x = module(x)
2025-09-07T08:03:06.8678169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8678504Z     out = self.relu(out)
2025-09-07T08:03:06.8678597Z 
2025-09-07T08:03:06.8678694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8679141Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8679566Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8679889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8680213Z     x = module(x)
2025-09-07T08:03:06.8680494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8680833Z     out = self.conv1(x)
2025-09-07T08:03:06.8680932Z 
2025-09-07T08:03:06.8681022Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8681475Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8681909Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8682221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8682547Z     x = module(x)
2025-09-07T08:03:06.8682840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8683173Z     out = self.relu(out)
2025-09-07T08:03:06.8683264Z 
2025-09-07T08:03:06.8683356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8683814Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8684341Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8684667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8684990Z     x = module(x)
2025-09-07T08:03:06.8685295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8685640Z     out = self.conv2(out)
2025-09-07T08:03:06.8685745Z 
2025-09-07T08:03:06.8685838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8686293Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8686734Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8687044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8687372Z     x = module(x)
2025-09-07T08:03:06.8687736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8688076Z     out = self.relu(out)
2025-09-07T08:03:06.8688171Z 
2025-09-07T08:03:06.8688260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8688716Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8689142Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8689461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8689781Z     x = module(x)
2025-09-07T08:03:06.8690066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8690399Z     out = self.conv3(out)
2025-09-07T08:03:06.8690501Z 
2025-09-07T08:03:06.8690590Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8691047Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8691477Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8691787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8692112Z     x = module(x)
2025-09-07T08:03:06.8692404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8692734Z     out = self.relu(out)
2025-09-07T08:03:06.8692827Z 
2025-09-07T08:03:06.8692917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8693368Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8693796Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8694120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8694442Z     x = module(x)
2025-09-07T08:03:06.8694723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8695054Z     out = self.conv1(x)
2025-09-07T08:03:06.8695153Z 
2025-09-07T08:03:06.8695246Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8695701Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8696121Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8696437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8696763Z     x = module(x)
2025-09-07T08:03:06.8697053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8697462Z     out = self.relu(out)
2025-09-07T08:03:06.8697555Z 
2025-09-07T08:03:06.8697647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8698142Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8698570Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8698889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8699210Z     x = module(x)
2025-09-07T08:03:06.8699494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8699826Z     out = self.conv2(out)
2025-09-07T08:03:06.8699923Z 
2025-09-07T08:03:06.8700024Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8700541Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8700973Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8701297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8701621Z     x = module(x)
2025-09-07T08:03:06.8701909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8702244Z     out = self.relu(out)
2025-09-07T08:03:06.8702336Z 
2025-09-07T08:03:06.8702428Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8702878Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8703433Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8703760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8704074Z     x = module(x)
2025-09-07T08:03:06.8704374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8704710Z     out = self.conv3(out)
2025-09-07T08:03:06.8704811Z 
2025-09-07T08:03:06.8704916Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8705372Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8705795Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8706114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8706439Z     x = module(x)
2025-09-07T08:03:06.8706731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8707063Z     out = self.relu(out)
2025-09-07T08:03:06.8707163Z 
2025-09-07T08:03:06.8707257Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8707717Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8708143Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8708462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8708777Z     x = module(x)
2025-09-07T08:03:06.8709072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8709404Z     out = self.conv1(x)
2025-09-07T08:03:06.8709498Z 
2025-09-07T08:03:06.8709597Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8710051Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8710472Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8710903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8711223Z     x = module(x)
2025-09-07T08:03:06.8711512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8711841Z     out = self.relu(out)
2025-09-07T08:03:06.8711945Z 
2025-09-07T08:03:06.8712036Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8712487Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8712918Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8713236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8713554Z     x = module(x)
2025-09-07T08:03:06.8713847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8714275Z     out = self.conv2(out)
2025-09-07T08:03:06.8714376Z 
2025-09-07T08:03:06.8714476Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8714930Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8715355Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8715674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8716000Z     x = module(x)
2025-09-07T08:03:06.8716291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8716616Z     out = self.relu(out)
2025-09-07T08:03:06.8716717Z 
2025-09-07T08:03:06.8716809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8717290Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8717739Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8718085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8718415Z     x = module(x)
2025-09-07T08:03:06.8718720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8719058Z     out = self.conv3(out)
2025-09-07T08:03:06.8719161Z 
2025-09-07T08:03:06.8719269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8719729Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8720154Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8720479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8720796Z     x = module(x)
2025-09-07T08:03:06.8721098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T08:03:06.8721437Z     identity = self.downsample(x)
2025-09-07T08:03:06.8721564Z 
2025-09-07T08:03:06.8721657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8722112Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8722545Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8722860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8723177Z     x = module(x)
2025-09-07T08:03:06.8723484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8723821Z     out = self.conv1(x)
2025-09-07T08:03:06.8723920Z 
2025-09-07T08:03:06.8724017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8724550Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8724977Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8725300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8725624Z     x = module(x)
2025-09-07T08:03:06.8725935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8726261Z     out = self.relu(out)
2025-09-07T08:03:06.8726371Z 
2025-09-07T08:03:06.8726466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8726921Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8727352Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8727738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8728060Z     x = module(x)
2025-09-07T08:03:06.8728357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8728692Z     out = self.conv2(out)
2025-09-07T08:03:06.8728794Z 
2025-09-07T08:03:06.8728898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8729347Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8729786Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8730105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8730428Z     x = module(x)
2025-09-07T08:03:06.8739096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8739529Z     out = self.relu(out)
2025-09-07T08:03:06.8739652Z 
2025-09-07T08:03:06.8739760Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8740272Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8740752Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8741112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8741445Z     x = module(x)
2025-09-07T08:03:06.8741758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8742103Z     out = self.conv3(out)
2025-09-07T08:03:06.8742208Z 
2025-09-07T08:03:06.8742316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8742786Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8743224Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8743556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8743874Z     x = module(x)
2025-09-07T08:03:06.8744166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8744490Z     out = self.relu(out)
2025-09-07T08:03:06.8744596Z 
2025-09-07T08:03:06.8744693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8745149Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8745575Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8745899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8746331Z     x = module(x)
2025-09-07T08:03:06.8746633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T08:03:06.8746967Z     out = self.conv1(x)
2025-09-07T08:03:06.8747064Z 
2025-09-07T08:03:06.8747165Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8747622Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8748064Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8748393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8748712Z     x = module(x)
2025-09-07T08:03:06.8749000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T08:03:06.8749324Z     out = self.relu(out)
2025-09-07T08:03:06.8749427Z 
2025-09-07T08:03:06.8749522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8750058Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8750494Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8750807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8751130Z     x = module(x)
2025-09-07T08:03:06.8751420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T08:03:06.8751754Z     out = self.conv2(out)
2025-09-07T08:03:06.8751853Z 
2025-09-07T08:03:06.8751948Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8752397Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8752818Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8753139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8753456Z     x = module(x)
2025-09-07T08:03:06.8753738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T08:03:06.8754066Z     out = self.relu(out)
2025-09-07T08:03:06.8754166Z 
2025-09-07T08:03:06.8754255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8754703Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8755141Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8755447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8755767Z     x = module(x)
2025-09-07T08:03:06.8756086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T08:03:06.8756419Z     out = self.conv3(out)
2025-09-07T08:03:06.8756514Z 
2025-09-07T08:03:06.8756607Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8757050Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T08:03:06.8757471Z     feats = self.feat_extractor(x)
2025-09-07T08:03:06.8757781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T08:03:06.8758097Z     x = module(x)
2025-09-07T08:03:06.8758373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T08:03:06.8758712Z     out = self.relu(out)
2025-09-07T08:03:06.8758811Z 
2025-09-07T08:03:06.8758902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8759358Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8759880Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8760297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8760803Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8761316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8761813Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8762003Z 
2025-09-07T08:03:06.8762103Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8762548Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8762981Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8763470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8763967Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8764464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8764951Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8765142Z 
2025-09-07T08:03:06.8765236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8765689Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8766115Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8766525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8767013Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8767503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8767992Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8768176Z 
2025-09-07T08:03:06.8768276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8768740Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8769169Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8769590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 86, in forward
2025-09-07T08:03:06.8770043Z     out.append(self.upsample(out[-1]) + t)
2025-09-07T08:03:06.8770178Z 
2025-09-07T08:03:06.8770279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8770727Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8771149Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8771561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8772048Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8772547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8773036Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8773222Z 
2025-09-07T08:03:06.8773392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8773841Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8774258Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8774662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 86, in forward
2025-09-07T08:03:06.8775091Z     out.append(self.upsample(out[-1]) + t)
2025-09-07T08:03:06.8775219Z 
2025-09-07T08:03:06.8775308Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8775753Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8776177Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8776576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T08:03:06.8777120Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8777610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T08:03:06.8778156Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T08:03:06.8778347Z 
2025-09-07T08:03:06.8778444Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8778900Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8779327Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8779737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 86, in forward
2025-09-07T08:03:06.8780168Z     out.append(self.upsample(out[-1]) + t)
2025-09-07T08:03:06.8780306Z 
2025-09-07T08:03:06.8780400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8780847Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8781265Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8781667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8782170Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8782666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8783161Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8783329Z 
2025-09-07T08:03:06.8783424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8783876Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8784299Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8784703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8785168Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8785638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8786106Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8786275Z 
2025-09-07T08:03:06.8786373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8786826Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8787329Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8787736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8788203Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8788668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8789140Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8789307Z 
2025-09-07T08:03:06.8789398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8789838Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8790256Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8790722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8791189Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8791659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8792129Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8792293Z 
2025-09-07T08:03:06.8792387Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8792858Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8793276Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8793679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8794145Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8794611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8795077Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8795242Z 
2025-09-07T08:03:06.8795331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8795770Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8796189Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8796589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8797045Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8797516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8797982Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8798157Z 
2025-09-07T08:03:06.8798246Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8798690Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8799108Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8799507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8799966Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8800431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8800993Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8801165Z 
2025-09-07T08:03:06.8801255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8801698Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8802117Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8802513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8804955Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8805497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8806251Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8806433Z 
2025-09-07T08:03:06.8806540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8807003Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8807434Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8807850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8808325Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8808802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8809270Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8809439Z 
2025-09-07T08:03:06.8809532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8810004Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8810443Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8810856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8811335Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8811817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8812291Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8812458Z 
2025-09-07T08:03:06.8812556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8813011Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8813435Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8813842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8814306Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8814777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8815253Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8815418Z 
2025-09-07T08:03:06.8815509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8815951Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8816483Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8816901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T08:03:06.8817378Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8817910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T08:03:06.8818383Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T08:03:06.8818553Z 
2025-09-07T08:03:06.8818646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8819097Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T08:03:06.8819518Z     feat_concat = self.fpn(feats)
2025-09-07T08:03:06.8819986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 91, in forward
2025-09-07T08:03:06.8820416Z     return torch.cat(out, dim=1)
2025-09-07T08:03:06.8820532Z 
2025-09-07T08:03:06.8820608Z cudagraph partition due to non gpu ops
2025-09-07T08:03:06.8820833Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8821282Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8821708Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8821838Z 
2025-09-07T08:03:06.8821930Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8822376Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8822799Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8822920Z 
2025-09-07T08:03:06.8823012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8823455Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8823877Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8823998Z 
2025-09-07T08:03:06.8824085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8824523Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T08:03:06.8824947Z     logits = self.prob_head(feat_concat)
2025-09-07T08:03:06.8825063Z 
2025-09-07T08:03:06.8825133Z cudagraph partition due to non gpu ops
2025-09-07T08:03:06.8825345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:06.8825803Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 202, in forward
2025-09-07T08:03:06.8826252Z     prob_map = _bf16_to_float32(torch.sigmoid(logits))
2025-09-07T08:03:06.8826641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/utils/pytorch.py", line 30, in _bf16_to_float32
2025-09-07T08:03:06.8827028Z     return x.float() if x.dtype == torch.bfloat16 else x
2025-09-07T08:03:06.8827178Z 
2025-09-07T08:03:40.9132332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:40.9132853Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/core.py", line 95, in __call__
2025-09-07T08:03:40.9133302Z     for bmap in (proba_map >= self.bin_thresh).astype(np.uint8)
2025-09-07T08:03:40.9133502Z 
2025-09-07T08:03:41.6699624Z pass
2025-09-07T08:03:41.6703296Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:03:44.5842662Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:03:44.5844097Z   import pynvml  # type: ignore[import]
2025-09-07T08:03:46.5809343Z 
2025-09-07T08:03:49.1854356Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:03:49.2282881Z 
2025-09-07T08:03:49.2283493Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:03:49.2283929Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:03:49.2285460Z cpu  eval  doctr_reco_predictor               
2025-09-07T08:03:49.7717396Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:03:49.7717974Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:03:49.8322440Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:03:49.8785153Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:03:58.5438771Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5439312Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5439752Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5439891Z 
2025-09-07T08:03:58.5440009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5440457Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5440861Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5441012Z 
2025-09-07T08:03:58.5441113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5441529Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5441962Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5442081Z 
2025-09-07T08:03:58.5442185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5442622Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5443001Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5443117Z 
2025-09-07T08:03:58.5443211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5443612Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5444003Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5444130Z 
2025-09-07T08:03:58.5444233Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5444637Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5445016Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5445145Z 
2025-09-07T08:03:58.5445239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5445632Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5446009Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5446128Z 
2025-09-07T08:03:58.5446224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5446613Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5446990Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5447111Z 
2025-09-07T08:03:58.5447200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5447597Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5447966Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5448501Z 
2025-09-07T08:03:58.5448598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5448995Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5449367Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5449482Z 
2025-09-07T08:03:58.5449577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5449961Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5450333Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5450456Z 
2025-09-07T08:03:58.5450544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5450936Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5451309Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5451429Z 
2025-09-07T08:03:58.5451648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5452043Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5452412Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5452531Z 
2025-09-07T08:03:58.5452630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5453021Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5453389Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5453511Z 
2025-09-07T08:03:58.5453602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5453998Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5454367Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5454484Z 
2025-09-07T08:03:58.5454583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5454971Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5455341Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5455464Z 
2025-09-07T08:03:58.5455555Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5455948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5456317Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5456435Z 
2025-09-07T08:03:58.5456523Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5456915Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5457284Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5457401Z 
2025-09-07T08:03:58.5457503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5457948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5458324Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5458464Z 
2025-09-07T08:03:58.5458556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5458948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5459319Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5459434Z 
2025-09-07T08:03:58.5459523Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5459910Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5460282Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5460397Z 
2025-09-07T08:03:58.5460493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5460974Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5461337Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5461461Z 
2025-09-07T08:03:58.5461550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5461938Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5462312Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5462428Z 
2025-09-07T08:03:58.5462525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5462911Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5463286Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5463407Z 
2025-09-07T08:03:58.5463496Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5463955Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5464335Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5464452Z 
2025-09-07T08:03:58.5464542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5464929Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5465300Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5465415Z 
2025-09-07T08:03:58.5465508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5465897Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5466264Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5466384Z 
2025-09-07T08:03:58.5466471Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5466875Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5467241Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5467357Z 
2025-09-07T08:03:58.5467447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5467840Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5468209Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5468324Z 
2025-09-07T08:03:58.5468421Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5468805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5469171Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5469293Z 
2025-09-07T08:03:58.5469379Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5469776Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5470148Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5470263Z 
2025-09-07T08:03:58.5470360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5470746Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5471120Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5471240Z 
2025-09-07T08:03:58.5471330Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5471717Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5472092Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5472210Z 
2025-09-07T08:03:58.5472299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5472706Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5473165Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5473280Z 
2025-09-07T08:03:58.5473380Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5473769Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5474149Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5474277Z 
2025-09-07T08:03:58.5474366Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5474762Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5475133Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5475249Z 
2025-09-07T08:03:58.5475339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5475795Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5476178Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5476294Z 
2025-09-07T08:03:58.5476392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5476783Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5477150Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5477274Z 
2025-09-07T08:03:58.5477363Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5477753Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5478127Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5478244Z 
2025-09-07T08:03:58.5478340Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5478739Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5479124Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5479244Z 
2025-09-07T08:03:58.5479333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5479724Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5480096Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5480213Z 
2025-09-07T08:03:58.5480303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5480692Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5481063Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5481179Z 
2025-09-07T08:03:58.5481277Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5481660Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5482038Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5482158Z 
2025-09-07T08:03:58.5482270Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5482658Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5483031Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5483148Z 
2025-09-07T08:03:58.5483245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5483629Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5484001Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5484124Z 
2025-09-07T08:03:58.5484214Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5484604Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5485063Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5485181Z 
2025-09-07T08:03:58.5485274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5485667Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5486041Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5486156Z 
2025-09-07T08:03:58.5486257Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5486643Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5487009Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5487135Z 
2025-09-07T08:03:58.5487227Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5487622Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5488013Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5488221Z 
2025-09-07T08:03:58.5488316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5488714Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5489083Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5489201Z 
2025-09-07T08:03:58.5489300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5489690Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5490054Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5490177Z 
2025-09-07T08:03:58.5490267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5490660Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5491029Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5491146Z 
2025-09-07T08:03:58.5491245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5491629Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5491998Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5492121Z 
2025-09-07T08:03:58.5492211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5492602Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5492971Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5493089Z 
2025-09-07T08:03:58.5493179Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5493565Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5493934Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5494053Z 
2025-09-07T08:03:58.5494154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5494547Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5494915Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5495037Z 
2025-09-07T08:03:58.5495127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5495516Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5495891Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5496007Z 
2025-09-07T08:03:58.5496099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:03:58.5496491Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T08:03:58.5496867Z     features = self.feat_extractor(x)
2025-09-07T08:03:58.5496982Z 
2025-09-07T08:04:07.6481452Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:07.6481997Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 75, in ctc_best_path
2025-09-07T08:04:07.6482508Z     probs = F.softmax(logits, dim=-1).max(dim=-1).values.min(dim=1).values
2025-09-07T08:04:07.6482709Z 
2025-09-07T08:04:07.8450419Z pass
2025-09-07T08:04:07.8454672Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:10.0531609Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:04:10.0532449Z   import pynvml  # type: ignore[import]
2025-09-07T08:04:12.0572511Z 
2025-09-07T08:04:12.1101900Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T08:04:12.1102597Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T08:04:12.1103373Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T08:04:12.1103971Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T08:04:13.0011944Z 
2025-09-07T08:04:13.0012298Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:04:13.0026510Z cpu  eval  drq                                
2025-09-07T08:04:13.0204981Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:13.0311008Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:13.0388586Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:15.0362267Z cudagraph partition due to non gpu ops
2025-09-07T08:04:15.0362561Z cudagraph partition due to non gpu ops
2025-09-07T08:04:15.0362783Z cudagraph partition due to non gpu ops
2025-09-07T08:04:15.0362976Z cudagraph partition due to non gpu ops
2025-09-07T08:04:15.0363216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0363602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0363949Z     return mod(*inputs)
2025-09-07T08:04:15.0364211Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0364522Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0364817Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T08:04:15.0365092Z     h = self.forward_conv(obs)
2025-09-07T08:04:15.0365389Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 43, in forward_conv
2025-09-07T08:04:15.0365693Z     conv = torch.relu(self.convs[0](obs))
2025-09-07T08:04:15.0365830Z 
2025-09-07T08:04:15.0365935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0366300Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0366624Z     return mod(*inputs)
2025-09-07T08:04:15.0366853Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0367140Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0367411Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T08:04:15.0367672Z     h = self.forward_conv(obs)
2025-09-07T08:04:15.0367928Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T08:04:15.0368211Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T08:04:15.0368340Z 
2025-09-07T08:04:15.0370258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0370650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0370985Z     return mod(*inputs)
2025-09-07T08:04:15.0371220Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0371509Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0371787Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T08:04:15.0372049Z     h = self.forward_conv(obs)
2025-09-07T08:04:15.0372301Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T08:04:15.0372590Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T08:04:15.0372724Z 
2025-09-07T08:04:15.0372845Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0373207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0373529Z     return mod(*inputs)
2025-09-07T08:04:15.0373914Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0374192Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0374467Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T08:04:15.0374746Z     h = self.forward_conv(obs)
2025-09-07T08:04:15.0375003Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T08:04:15.0375283Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T08:04:15.0375411Z 
2025-09-07T08:04:15.0375511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0375861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0376175Z     return mod(*inputs)
2025-09-07T08:04:15.0376414Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0376701Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0376992Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T08:04:15.0377247Z     h = self.forward_conv(obs)
2025-09-07T08:04:15.0377515Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T08:04:15.0377873Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T08:04:15.0377999Z 
2025-09-07T08:04:15.0378102Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0378448Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0378768Z     return mod(*inputs)
2025-09-07T08:04:15.0378996Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0379275Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0379546Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 61, in forward
2025-09-07T08:04:15.0379793Z     out = self.head(h)
2025-09-07T08:04:15.0379896Z 
2025-09-07T08:04:15.0379999Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0380350Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0380666Z     return mod(*inputs)
2025-09-07T08:04:15.0380897Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T08:04:15.0381189Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T08:04:15.0381476Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 63, in forward
2025-09-07T08:04:15.0381743Z     out = torch.tanh(out)
2025-09-07T08:04:15.0381847Z 
2025-09-07T08:04:15.0381950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0382292Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0382609Z     return mod(*inputs)
2025-09-07T08:04:15.0382847Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T08:04:15.0383133Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T08:04:15.0383362Z 
2025-09-07T08:04:15.0383458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0383803Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0384117Z     return mod(*inputs)
2025-09-07T08:04:15.0384350Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T08:04:15.0384629Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T08:04:15.0384766Z 
2025-09-07T08:04:15.0384859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0385202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0385519Z     return mod(*inputs)
2025-09-07T08:04:15.0385753Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T08:04:15.0386026Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T08:04:15.0386171Z 
2025-09-07T08:04:15.0386332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0386681Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0386998Z     return mod(*inputs)
2025-09-07T08:04:15.0387228Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T08:04:15.0387497Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T08:04:15.0387638Z 
2025-09-07T08:04:15.0387730Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0388069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0388396Z     return mod(*inputs)
2025-09-07T08:04:15.0388637Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T08:04:15.0388930Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T08:04:15.0389082Z 
2025-09-07T08:04:15.0389185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:15.0389544Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:15.0389863Z     return mod(*inputs)
2025-09-07T08:04:15.0390098Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 105, in forward
2025-09-07T08:04:15.0390365Z     std = log_std.exp()
2025-09-07T08:04:15.0390457Z 
2025-09-07T08:04:22.6632515Z pass
2025-09-07T08:04:22.6635152Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:24.6637802Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:04:24.6638777Z   import pynvml  # type: ignore[import]
2025-09-07T08:04:26.6769725Z 
2025-09-07T08:04:28.2404632Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:04:28.2405127Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:04:28.2595646Z cpu  eval  fastNLP_Bert                       
2025-09-07T08:04:29.2642669Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:29.6611759Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:30.1199397Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:30.2478524Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:04:30.2479192Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:04:30.2479700Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T08:04:30.2480225Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:04:30.2481212Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T08:04:30.2481725Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:04:30.2482179Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T08:04:30.2482842Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:30.2483517Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(*inputs)
2025-09-07T08:04:30.2484328Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T08:04:30.2485008Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     sequence_output = self.bert(words)
2025-09-07T08:04:30.2485718Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T08:04:30.2486443Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.model(words)
2025-09-07T08:04:30.2487124Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T08:04:30.2490991Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T08:04:30.2491622Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:04:30.2492003Z W0907 08:04:30.247199 41954 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:04:37.1479308Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:37.1479818Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T08:04:37.1480759Z     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T08:04:37.1481039Z 
2025-09-07T08:04:37.1481151Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:37.1481589Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 441, in forward
2025-09-07T08:04:37.1481974Z     seq_len = word_mask.sum(dim=-1)
2025-09-07T08:04:37.1482147Z 
2025-09-07T08:04:42.0582240Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0582742Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward
2025-09-07T08:04:42.0583180Z     embedding_output = self.embeddings(input_ids, token_type_ids)
2025-09-07T08:04:42.0583605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 243, in forward
2025-09-07T08:04:42.0583991Z     embeddings = self.LayerNorm(embeddings)
2025-09-07T08:04:42.0584138Z 
2025-09-07T08:04:42.0584243Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0584636Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0585019Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0585394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0586258Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0586682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0587121Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0587540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0587924Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0588305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0588675Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0588823Z 
2025-09-07T08:04:42.0588934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0589316Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0589819Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0590198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0590596Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0590991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0591394Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0591786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0592159Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0592530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0592912Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0593053Z 
2025-09-07T08:04:42.0593160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0593525Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0593889Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0594266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0594657Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0595031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0595432Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0595830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0596198Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0596574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0596934Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0597081Z 
2025-09-07T08:04:42.0597158Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0597363Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0597669Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0598047Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0598410Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0598769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0599154Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0599534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0600011Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0600410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0600802Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0601185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0601552Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0601684Z 
2025-09-07T08:04:42.0601782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0602144Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0602511Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0602933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0603479Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0603864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0604282Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0604674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0605040Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0605174Z 
2025-09-07T08:04:42.0605278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0605664Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0606042Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0606428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0606829Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0607210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0607608Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0608000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0608390Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0608757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0609129Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0609288Z 
2025-09-07T08:04:42.0609388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0609774Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0610150Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0610525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0610930Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0611316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0611726Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0612132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0612483Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0612625Z 
2025-09-07T08:04:42.0612722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0613220Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0613589Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0613946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0614332Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0614718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0615126Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0615530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0615906Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0616285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0616748Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0616897Z 
2025-09-07T08:04:42.0617005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0617381Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0617799Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0618158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0618548Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0629524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0630038Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0630472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0630878Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0631252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0631619Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0631769Z 
2025-09-07T08:04:42.0631875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0632277Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0632662Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0633025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0633422Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0633824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0634250Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0634655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0635036Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0635415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0635783Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0635928Z 
2025-09-07T08:04:42.0636016Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0636221Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0636438Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0636819Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0637192Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0637668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0638050Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0638438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0638842Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0639249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0639648Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0640040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0640411Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0640560Z 
2025-09-07T08:04:42.0640727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0641121Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0641498Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0641861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0642253Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0642642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0643042Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0643422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0643780Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0643917Z 
2025-09-07T08:04:42.0644016Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0644391Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0644761Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0645113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0645498Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0645877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0646273Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0646654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0647034Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0647420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0647795Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0647945Z 
2025-09-07T08:04:42.0648047Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0648426Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0648785Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0649145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0649527Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0649911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0650320Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0650810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0651178Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0651315Z 
2025-09-07T08:04:42.0651424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0651810Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0652198Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0652577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0652978Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0653372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0653782Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0654244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0654634Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0655016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0655385Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0655532Z 
2025-09-07T08:04:42.0655640Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0656010Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0656378Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0656743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0657128Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0657508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0658007Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0658413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0658791Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0659152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0659515Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0659649Z 
2025-09-07T08:04:42.0659752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0660123Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0660484Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0660852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0661233Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0661614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0662004Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0662400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0662773Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0663140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0663499Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0663638Z 
2025-09-07T08:04:42.0663715Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0663995Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0664216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0664584Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0664946Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0665295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0665676Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0666056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0666452Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0666839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0667319Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0667711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0668070Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0668205Z 
2025-09-07T08:04:42.0668308Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0668678Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0669042Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0669403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0669785Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0670162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0670548Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0670932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0671285Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0671415Z 
2025-09-07T08:04:42.0671515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0671887Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0672247Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0672605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0672985Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0673364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0673750Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0674133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0674517Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0674894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0675263Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0675409Z 
2025-09-07T08:04:42.0675502Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0675871Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0676236Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0676593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0677048Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0677420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0677828Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0678230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0678592Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0678723Z 
2025-09-07T08:04:42.0678821Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0679187Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0679552Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0679915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0680379Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0680757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0681161Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0681559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0681933Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0682308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0682666Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0682819Z 
2025-09-07T08:04:42.0682913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0683280Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0683652Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0684012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0684391Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0684767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0685167Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0685564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0685939Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0686314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0686666Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0686802Z 
2025-09-07T08:04:42.0686896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0687263Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0687623Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0687981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0688358Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0688737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0689131Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0689523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0689893Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0690331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0690693Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0690838Z 
2025-09-07T08:04:42.0690919Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0691123Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0691334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0691699Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0692065Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0692425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0692806Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0693243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0693647Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0694042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0694437Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0694837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0695202Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0695344Z 
2025-09-07T08:04:42.0695440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0695811Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0696178Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0696542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0696924Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0697304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0697741Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0698126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0698478Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0698606Z 
2025-09-07T08:04:42.0698697Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0699064Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0699427Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0699788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0700166Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0700544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0700926Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0701305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0701687Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0702057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0702423Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0702576Z 
2025-09-07T08:04:42.0702666Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0703227Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0704939Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0705293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0705684Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0706066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0706481Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0706893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0707259Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0707405Z 
2025-09-07T08:04:42.0707508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0707991Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0708379Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0708745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0709137Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0709524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0709936Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0710338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0710712Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0711085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0711457Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0711603Z 
2025-09-07T08:04:42.0711705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0712081Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0712444Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0712807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0713204Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0713588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0713991Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0714475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0714858Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0715231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0715589Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0715724Z 
2025-09-07T08:04:42.0715821Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0716189Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0716558Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0716920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0717302Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0717677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0718156Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0718552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0718923Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0719289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0719646Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0719798Z 
2025-09-07T08:04:42.0719876Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0720079Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0720297Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0720661Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0721026Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0721447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0721838Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0722222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0722616Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0723012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0723404Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0723802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0724162Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0724293Z 
2025-09-07T08:04:42.0724389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0724770Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0725134Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0725493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0725877Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0726246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0726635Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0727008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0727357Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0727484Z 
2025-09-07T08:04:42.0727573Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0727944Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0728298Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0728652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0729029Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0729427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0729833Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0730211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0730597Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0730971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0731429Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0731580Z 
2025-09-07T08:04:42.0731670Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0732037Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0732402Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0732773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0733146Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0733516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0733917Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0734381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0734730Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0734867Z 
2025-09-07T08:04:42.0734963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0735349Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0735706Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0736057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0736425Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0736794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0737191Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0737721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0738099Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0738462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0738820Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0738970Z 
2025-09-07T08:04:42.0739064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0739441Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0739801Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0740151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0740525Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0740901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0741295Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0741684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0742044Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0742419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0742765Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0742893Z 
2025-09-07T08:04:42.0742986Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0743344Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0743700Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0744133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0744524Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0744895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0745287Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0745677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0746040Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0746402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0746759Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0746901Z 
2025-09-07T08:04:42.0746973Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0747233Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0747442Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0747817Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0748167Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0748515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0748889Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0749259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0749641Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0750021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0750408Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0750784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0751126Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0751255Z 
2025-09-07T08:04:42.0751359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0751719Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0752069Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0752424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0752827Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0753218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0753597Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0753988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0754342Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0754468Z 
2025-09-07T08:04:42.0754566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0754925Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0755290Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0755646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0756043Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0756435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0756825Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0757290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0757671Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0758050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0758420Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0758569Z 
2025-09-07T08:04:42.0758665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0759040Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0759406Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0759768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0760221Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0760598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0761000Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0761398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0761752Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0761882Z 
2025-09-07T08:04:42.0761975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0762342Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0762704Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0763062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0763451Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0763818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0764224Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0764621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0764999Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0765370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0765726Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0765873Z 
2025-09-07T08:04:42.0765975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0766369Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0766737Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0767093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0767469Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0767843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0768244Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0768636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0769003Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0769373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0769731Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0769928Z 
2025-09-07T08:04:42.0770029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0770395Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0770750Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0771110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0771492Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0771873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0772273Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0772657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0773023Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0773453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0773831Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0773971Z 
2025-09-07T08:04:42.0774052Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0774247Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0774465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0774834Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0775203Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0775562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0775946Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0776326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0776727Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0777120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0777501Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0777974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0778330Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0778461Z 
2025-09-07T08:04:42.0778562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0778930Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0779286Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0779651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0780035Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0780424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0780811Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0781197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0781552Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0781683Z 
2025-09-07T08:04:42.0781782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0782154Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0782511Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0782870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0783342Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0783720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0784108Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0784482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0784864Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0785237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0785605Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0785752Z 
2025-09-07T08:04:42.0785849Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0786340Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0786714Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0787073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0787446Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0787819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0788219Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0788620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0788980Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0789113Z 
2025-09-07T08:04:42.0789210Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0789570Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0789939Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0790295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0790676Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0791055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0791451Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0791846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0792217Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0792587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0792953Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0793097Z 
2025-09-07T08:04:42.0793195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0793563Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0793923Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0794280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0794658Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0795030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0795429Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0795824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0796273Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0796638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0796993Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0797128Z 
2025-09-07T08:04:42.0797220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0797587Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0797958Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0798311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0798708Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0799086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0799546Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0799951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0800322Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0800692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0801055Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0801195Z 
2025-09-07T08:04:42.0801276Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0801473Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0801688Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0802054Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0802414Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0802789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0803303Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0803682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0804080Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0804476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0804864Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0805239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0805593Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0805730Z 
2025-09-07T08:04:42.0805881Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0806254Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0806614Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0806961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0807341Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0807713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0808107Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0808484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0808834Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0808970Z 
2025-09-07T08:04:42.0809057Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0809538Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0809902Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0810253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0810638Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0811015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0811405Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0811787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0812158Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0812627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0813003Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0813153Z 
2025-09-07T08:04:42.0813251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0813622Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0813982Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0814341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0814718Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0815101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0815507Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0815902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0816266Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0816404Z 
2025-09-07T08:04:42.0816502Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0816873Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0817230Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0817646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0818033Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0818414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0818817Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0819212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0819586Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0819956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0820325Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0820466Z 
2025-09-07T08:04:42.0820566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0820929Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0821291Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0821652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0822035Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0822497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0822895Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0823292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0823665Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0824033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0824378Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0824515Z 
2025-09-07T08:04:42.0824606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0824990Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0825370Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0825824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0826221Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0826615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0827026Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0827425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0827822Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0828188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0828555Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0828704Z 
2025-09-07T08:04:42.0828784Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0828991Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0829212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0829587Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0829960Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0830339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0830740Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0831122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0831528Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0831930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0832328Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0832720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0833075Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0833216Z 
2025-09-07T08:04:42.0833314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0833705Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0834073Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0834432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0834814Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0835199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0835600Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0836090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0836443Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0836588Z 
2025-09-07T08:04:42.0836682Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0837051Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0837419Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0837782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0838163Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0838546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0838935Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0839383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0839792Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0840170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0840545Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0840702Z 
2025-09-07T08:04:42.0840798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0841173Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0841539Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0841895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0842282Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0842671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0843077Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0843471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0843829Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0843969Z 
2025-09-07T08:04:42.0844059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0844424Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0844795Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0845147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0845534Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0845913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0846319Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0846717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0847087Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0847463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0847828Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0847973Z 
2025-09-07T08:04:42.0848069Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0848440Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0848886Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0849245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0849630Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0850011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0850404Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0850798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0851167Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0851539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0851892Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0852025Z 
2025-09-07T08:04:42.0852178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0852549Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0852909Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0853266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0853647Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0854020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0854413Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0854806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0855179Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0855556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0855909Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0856052Z 
2025-09-07T08:04:42.0856127Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0856327Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0856545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0856908Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0857269Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0857689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0858074Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0858455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0858851Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0859243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0859631Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0860008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0860364Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0860495Z 
2025-09-07T08:04:42.0860586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0860954Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0861314Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0861673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0862157Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0862539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0862931Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0863318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0863672Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0863804Z 
2025-09-07T08:04:42.0863896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0864264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0864623Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0864981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0865423Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0865801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0866186Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0866576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0866956Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0867336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0867699Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0867854Z 
2025-09-07T08:04:42.0867945Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0868317Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0868680Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0869032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0869413Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0869793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0870195Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0870592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0870939Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0871078Z 
2025-09-07T08:04:42.0871169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0871537Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0871924Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0872284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0872655Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0873037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0873438Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0873839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0874211Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0874575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0875017Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0875163Z 
2025-09-07T08:04:42.0875255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0875620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0875981Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0876331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0876713Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0877090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0877485Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0877879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0878333Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0878717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0879078Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0879213Z 
2025-09-07T08:04:42.0879316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0879688Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0880061Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0880424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0880814Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0881206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0881610Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0882005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0882381Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0882751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0883112Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0883255Z 
2025-09-07T08:04:42.0883329Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0883531Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0883752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0884120Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0884481Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0884845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0885233Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0885614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0886011Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0886400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0886791Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0887169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0887516Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0887645Z 
2025-09-07T08:04:42.0887744Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0888187Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0888553Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0888909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0889288Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0889657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0890040Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0890419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0890771Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0890901Z 
2025-09-07T08:04:42.0890997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0891425Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0891794Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0892147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0892526Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0892906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0893283Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0893660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0894037Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0894404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0894773Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0894921Z 
2025-09-07T08:04:42.0895013Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0895388Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0895750Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0896104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0896476Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0896851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0905749Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0906235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0906626Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0906768Z 
2025-09-07T08:04:42.0906880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0907274Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0907655Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0908023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0908418Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0908804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0909213Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0909618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0910180Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0910548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T08:04:42.0910900Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T08:04:42.0911044Z 
2025-09-07T08:04:42.0911144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0911514Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0911874Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0912226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0912607Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0913082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0913489Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0913883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0914251Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0914607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T08:04:42.0914957Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T08:04:42.0915091Z 
2025-09-07T08:04:42.0915182Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0915548Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0915905Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0916254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0916632Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0917007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0917397Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0917782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T08:04:42.0918146Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T08:04:42.0918504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T08:04:42.0918864Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T08:04:42.0919004Z 
2025-09-07T08:04:42.0919088Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0919286Z cudagraph partition due to non gpu ops
2025-09-07T08:04:42.0919505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0919873Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0920237Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0920584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0920961Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0921334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T08:04:42.0921725Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T08:04:42.0922115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T08:04:42.0922498Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T08:04:42.0922974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T08:04:42.0923345Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0923479Z 
2025-09-07T08:04:42.0923576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0923949Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0924304Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0924661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0925036Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0925421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0925805Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0926245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T08:04:42.0926601Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0926739Z 
2025-09-07T08:04:42.0926831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0927216Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0927571Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0927921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0928295Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0928667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T08:04:42.0929059Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:04:42.0929434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T08:04:42.0929811Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:04:42.0930189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T08:04:42.0930551Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T08:04:42.0930704Z 
2025-09-07T08:04:42.0930802Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0931173Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T08:04:42.0931542Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T08:04:42.0931900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T08:04:42.0932287Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T08:04:42.0932665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T08:04:42.0933064Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:04:42.0933461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T08:04:42.0933815Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:04:42.0933945Z 
2025-09-07T08:04:42.0934038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0934410Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 518, in forward
2025-09-07T08:04:42.0934773Z     pooled_output = self.pooler(sequence_output)
2025-09-07T08:04:42.0935133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 404, in forward
2025-09-07T08:04:42.0935515Z     pooled_output = self.dense(first_token_tensor)
2025-09-07T08:04:42.0935756Z 
2025-09-07T08:04:42.0935854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:42.0936225Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 518, in forward
2025-09-07T08:04:42.0936587Z     pooled_output = self.pooler(sequence_output)
2025-09-07T08:04:42.0936936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 405, in forward
2025-09-07T08:04:42.0937302Z     pooled_output = self.activation(pooled_output)
2025-09-07T08:04:42.0937446Z 
2025-09-07T08:04:44.1964142Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:44.1964722Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 140, in torch_dynamo_resume_in_forward_at_137
2025-09-07T08:04:44.1965189Z     return self.dropout(outputs)
2025-09-07T08:04:44.1965967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/embedding.py", line 160, in dropout
2025-09-07T08:04:44.1966358Z     return self.dropout_layer(words)
2025-09-07T08:04:44.1966497Z 
2025-09-07T08:04:44.2865852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:44.2866329Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 266, in torch_dynamo_resume_in_forward_at_265
2025-09-07T08:04:44.2866816Z     logits = self.qa_outputs(sequence_output)  # [batch_size, seq_len, num_labels]
2025-09-07T08:04:44.2867025Z 
2025-09-07T08:04:44.5107025Z pass
2025-09-07T08:04:44.5110279Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:46.7284807Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:04:46.7285716Z   import pynvml  # type: ignore[import]
2025-09-07T08:04:48.7399912Z 
2025-09-07T08:04:49.1702545Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:04:49.1702847Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:04:49.1739782Z cpu  eval  functorch_dp_cifar10               
2025-09-07T08:04:49.2401780Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:49.2675894Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:49.2900769Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:04:52.7216478Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7216787Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7217009Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7217216Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7217423Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7217728Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7217923Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7218104Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7218296Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7218504Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7218700Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7218896Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7219085Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7219282Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7219499Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7219692Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7219885Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7220066Z cudagraph partition due to non gpu ops
2025-09-07T08:04:52.7220301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7220693Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7221480Z     return mod(*inputs)
2025-09-07T08:04:52.7221836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7222200Z     return self._forward_impl(x)
2025-09-07T08:04:52.7222556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 268, in _forward_impl
2025-09-07T08:04:52.7222912Z     x = self.conv1(x)
2025-09-07T08:04:52.7223008Z 
2025-09-07T08:04:52.7223121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7223481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7223807Z     return mod(*inputs)
2025-09-07T08:04:52.7224121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7224467Z     return self._forward_impl(x)
2025-09-07T08:04:52.7224947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 270, in _forward_impl
2025-09-07T08:04:52.7225306Z     x = self.relu(x)
2025-09-07T08:04:52.7225404Z 
2025-09-07T08:04:52.7225497Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7225842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7226161Z     return mod(*inputs)
2025-09-07T08:04:52.7226462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7226800Z     return self._forward_impl(x)
2025-09-07T08:04:52.7227135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 271, in _forward_impl
2025-09-07T08:04:52.7227484Z     x = self.maxpool(x)
2025-09-07T08:04:52.7227572Z 
2025-09-07T08:04:52.7227667Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7228013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7228324Z     return mod(*inputs)
2025-09-07T08:04:52.7228626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7228963Z     return self._forward_impl(x)
2025-09-07T08:04:52.7229289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7229636Z     x = self.layer1(x)
2025-09-07T08:04:52.7229949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7230279Z     out = self.conv1(x)
2025-09-07T08:04:52.7230368Z 
2025-09-07T08:04:52.7230461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7230816Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7231127Z     return mod(*inputs)
2025-09-07T08:04:52.7231441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7231779Z     return self._forward_impl(x)
2025-09-07T08:04:52.7232111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7232460Z     x = self.layer1(x)
2025-09-07T08:04:52.7232759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7233097Z     out = self.relu(out)
2025-09-07T08:04:52.7233200Z 
2025-09-07T08:04:52.7233293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7233638Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7233949Z     return mod(*inputs)
2025-09-07T08:04:52.7234253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7234677Z     return self._forward_impl(x)
2025-09-07T08:04:52.7235012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7235355Z     x = self.layer1(x)
2025-09-07T08:04:52.7235652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7235985Z     out = self.conv2(out)
2025-09-07T08:04:52.7236086Z 
2025-09-07T08:04:52.7236185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7236518Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7236827Z     return mod(*inputs)
2025-09-07T08:04:52.7237128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7237463Z     return self._forward_impl(x)
2025-09-07T08:04:52.7237852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7238206Z     x = self.layer1(x)
2025-09-07T08:04:52.7238504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7238837Z     out = self.relu(out)
2025-09-07T08:04:52.7238937Z 
2025-09-07T08:04:52.7239037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7239366Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7239670Z     return mod(*inputs)
2025-09-07T08:04:52.7239967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7240297Z     return self._forward_impl(x)
2025-09-07T08:04:52.7240620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7240965Z     x = self.layer1(x)
2025-09-07T08:04:52.7241272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7241597Z     out = self.conv1(x)
2025-09-07T08:04:52.7241689Z 
2025-09-07T08:04:52.7241785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7242154Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7242464Z     return mod(*inputs)
2025-09-07T08:04:52.7242765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7243102Z     return self._forward_impl(x)
2025-09-07T08:04:52.7243453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7243809Z     x = self.layer1(x)
2025-09-07T08:04:52.7244123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7244463Z     out = self.relu(out)
2025-09-07T08:04:52.7244567Z 
2025-09-07T08:04:52.7244676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7245032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7245349Z     return mod(*inputs)
2025-09-07T08:04:52.7245664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7246013Z     return self._forward_impl(x)
2025-09-07T08:04:52.7246351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7246702Z     x = self.layer1(x)
2025-09-07T08:04:52.7247001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7247333Z     out = self.conv2(out)
2025-09-07T08:04:52.7247435Z 
2025-09-07T08:04:52.7247534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7247954Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7248266Z     return mod(*inputs)
2025-09-07T08:04:52.7248574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7248916Z     return self._forward_impl(x)
2025-09-07T08:04:52.7249247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T08:04:52.7249588Z     x = self.layer1(x)
2025-09-07T08:04:52.7249884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7250218Z     out = self.relu(out)
2025-09-07T08:04:52.7250314Z 
2025-09-07T08:04:52.7250412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7250744Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7251126Z     return mod(*inputs)
2025-09-07T08:04:52.7251430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7251762Z     return self._forward_impl(x)
2025-09-07T08:04:52.7252089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7252435Z     x = self.layer2(x)
2025-09-07T08:04:52.7252731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7253060Z     out = self.conv1(x)
2025-09-07T08:04:52.7253150Z 
2025-09-07T08:04:52.7253248Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7253576Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7253882Z     return mod(*inputs)
2025-09-07T08:04:52.7254178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7254516Z     return self._forward_impl(x)
2025-09-07T08:04:52.7254838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7255180Z     x = self.layer2(x)
2025-09-07T08:04:52.7255470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7255798Z     out = self.relu(out)
2025-09-07T08:04:52.7255891Z 
2025-09-07T08:04:52.7255989Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7256313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7256620Z     return mod(*inputs)
2025-09-07T08:04:52.7256917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7257247Z     return self._forward_impl(x)
2025-09-07T08:04:52.7257653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7257998Z     x = self.layer2(x)
2025-09-07T08:04:52.7258298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7258650Z     out = self.conv2(out)
2025-09-07T08:04:52.7258754Z 
2025-09-07T08:04:52.7258855Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7259205Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7259521Z     return mod(*inputs)
2025-09-07T08:04:52.7259837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7260176Z     return self._forward_impl(x)
2025-09-07T08:04:52.7260509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7260943Z     x = self.layer2(x)
2025-09-07T08:04:52.7261244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 100, in forward
2025-09-07T08:04:52.7261589Z     identity = self.downsample(x)
2025-09-07T08:04:52.7261707Z 
2025-09-07T08:04:52.7261809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7262145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7262457Z     return mod(*inputs)
2025-09-07T08:04:52.7262765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7263103Z     return self._forward_impl(x)
2025-09-07T08:04:52.7263435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7263783Z     x = self.layer2(x)
2025-09-07T08:04:52.7264101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7264515Z     out = self.relu(out)
2025-09-07T08:04:52.7264616Z 
2025-09-07T08:04:52.7264724Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7265069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7265392Z     return mod(*inputs)
2025-09-07T08:04:52.7265707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7266053Z     return self._forward_impl(x)
2025-09-07T08:04:52.7266393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7266742Z     x = self.layer2(x)
2025-09-07T08:04:52.7267050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7267382Z     out = self.conv1(x)
2025-09-07T08:04:52.7267475Z 
2025-09-07T08:04:52.7267582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7267924Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7268235Z     return mod(*inputs)
2025-09-07T08:04:52.7268540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7268876Z     return self._forward_impl(x)
2025-09-07T08:04:52.7269205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7269569Z     x = self.layer2(x)
2025-09-07T08:04:52.7269881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7270214Z     out = self.relu(out)
2025-09-07T08:04:52.7270313Z 
2025-09-07T08:04:52.7270412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7270750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7271062Z     return mod(*inputs)
2025-09-07T08:04:52.7271361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7271700Z     return self._forward_impl(x)
2025-09-07T08:04:52.7272029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7272369Z     x = self.layer2(x)
2025-09-07T08:04:52.7272662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7272992Z     out = self.conv2(out)
2025-09-07T08:04:52.7273096Z 
2025-09-07T08:04:52.7273194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7273523Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7273830Z     return mod(*inputs)
2025-09-07T08:04:52.7274232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7274566Z     return self._forward_impl(x)
2025-09-07T08:04:52.7274889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T08:04:52.7275243Z     x = self.layer2(x)
2025-09-07T08:04:52.7275542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7275873Z     out = self.relu(out)
2025-09-07T08:04:52.7275967Z 
2025-09-07T08:04:52.7276066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7276391Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7276700Z     return mod(*inputs)
2025-09-07T08:04:52.7276995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7277335Z     return self._forward_impl(x)
2025-09-07T08:04:52.7277725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7278080Z     x = self.layer3(x)
2025-09-07T08:04:52.7278383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7278708Z     out = self.conv1(x)
2025-09-07T08:04:52.7278798Z 
2025-09-07T08:04:52.7278892Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7279219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7279525Z     return mod(*inputs)
2025-09-07T08:04:52.7279820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7280154Z     return self._forward_impl(x)
2025-09-07T08:04:52.7280484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7280831Z     x = self.layer3(x)
2025-09-07T08:04:52.7281119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7281445Z     out = self.relu(out)
2025-09-07T08:04:52.7281536Z 
2025-09-07T08:04:52.7281631Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7281960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7282266Z     return mod(*inputs)
2025-09-07T08:04:52.7282559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7282891Z     return self._forward_impl(x)
2025-09-07T08:04:52.7283210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7283553Z     x = self.layer3(x)
2025-09-07T08:04:52.7283852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7284191Z     out = self.conv2(out)
2025-09-07T08:04:52.7284287Z 
2025-09-07T08:04:52.7284385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7284715Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7285027Z     return mod(*inputs)
2025-09-07T08:04:52.7285326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7285658Z     return self._forward_impl(x)
2025-09-07T08:04:52.7285981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7286321Z     x = self.layer3(x)
2025-09-07T08:04:52.7286615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 100, in forward
2025-09-07T08:04:52.7286956Z     identity = self.downsample(x)
2025-09-07T08:04:52.7287145Z 
2025-09-07T08:04:52.7287248Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7287578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7287905Z     return mod(*inputs)
2025-09-07T08:04:52.7288224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7288574Z     return self._forward_impl(x)
2025-09-07T08:04:52.7288919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7289277Z     x = self.layer3(x)
2025-09-07T08:04:52.7289587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7289924Z     out = self.relu(out)
2025-09-07T08:04:52.7290022Z 
2025-09-07T08:04:52.7290129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7290555Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7290879Z     return mod(*inputs)
2025-09-07T08:04:52.7291185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7291526Z     return self._forward_impl(x)
2025-09-07T08:04:52.7291857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7292202Z     x = self.layer3(x)
2025-09-07T08:04:52.7292501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7292830Z     out = self.conv1(x)
2025-09-07T08:04:52.7292922Z 
2025-09-07T08:04:52.7293019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7293355Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7293672Z     return mod(*inputs)
2025-09-07T08:04:52.7293972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7294307Z     return self._forward_impl(x)
2025-09-07T08:04:52.7294632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7294977Z     x = self.layer3(x)
2025-09-07T08:04:52.7295268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7295594Z     out = self.relu(out)
2025-09-07T08:04:52.7295686Z 
2025-09-07T08:04:52.7295783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7296110Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7296416Z     return mod(*inputs)
2025-09-07T08:04:52.7296715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7297052Z     return self._forward_impl(x)
2025-09-07T08:04:52.7297375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7297786Z     x = self.layer3(x)
2025-09-07T08:04:52.7298094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7298428Z     out = self.conv2(out)
2025-09-07T08:04:52.7298532Z 
2025-09-07T08:04:52.7298628Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7298964Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7299275Z     return mod(*inputs)
2025-09-07T08:04:52.7299649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7300010Z     return self._forward_impl(x)
2025-09-07T08:04:52.7300369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T08:04:52.7300808Z     x = self.layer3(x)
2025-09-07T08:04:52.7301120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7301461Z     out = self.relu(out)
2025-09-07T08:04:52.7301564Z 
2025-09-07T08:04:52.7301673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7302034Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7302357Z     return mod(*inputs)
2025-09-07T08:04:52.7302663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7303004Z     return self._forward_impl(x)
2025-09-07T08:04:52.7303538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7303889Z     x = self.layer4(x)
2025-09-07T08:04:52.7304298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7304642Z     out = self.conv1(x)
2025-09-07T08:04:52.7304736Z 
2025-09-07T08:04:52.7304839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7305180Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7305498Z     return mod(*inputs)
2025-09-07T08:04:52.7305796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7306137Z     return self._forward_impl(x)
2025-09-07T08:04:52.7306481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7306850Z     x = self.layer4(x)
2025-09-07T08:04:52.7307147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7307487Z     out = self.relu(out)
2025-09-07T08:04:52.7307582Z 
2025-09-07T08:04:52.7307684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7308012Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7308321Z     return mod(*inputs)
2025-09-07T08:04:52.7308622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7308957Z     return self._forward_impl(x)
2025-09-07T08:04:52.7309285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7309629Z     x = self.layer4(x)
2025-09-07T08:04:52.7309919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7310250Z     out = self.conv2(out)
2025-09-07T08:04:52.7310352Z 
2025-09-07T08:04:52.7310447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7310782Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7311088Z     return mod(*inputs)
2025-09-07T08:04:52.7311380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7311715Z     return self._forward_impl(x)
2025-09-07T08:04:52.7312042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7312385Z     x = self.layer4(x)
2025-09-07T08:04:52.7312682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 100, in forward
2025-09-07T08:04:52.7313024Z     identity = self.downsample(x)
2025-09-07T08:04:52.7313142Z 
2025-09-07T08:04:52.7313239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7313568Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7313997Z     return mod(*inputs)
2025-09-07T08:04:52.7314299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7314638Z     return self._forward_impl(x)
2025-09-07T08:04:52.7314971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7315316Z     x = self.layer4(x)
2025-09-07T08:04:52.7315614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T08:04:52.7315946Z     out = self.relu(out)
2025-09-07T08:04:52.7316044Z 
2025-09-07T08:04:52.7316143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7316471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7316777Z     return mod(*inputs)
2025-09-07T08:04:52.7317153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7317504Z     return self._forward_impl(x)
2025-09-07T08:04:52.7317832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7318178Z     x = self.layer4(x)
2025-09-07T08:04:52.7318473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T08:04:52.7318802Z     out = self.conv1(x)
2025-09-07T08:04:52.7318895Z 
2025-09-07T08:04:52.7318994Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7319322Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7319628Z     return mod(*inputs)
2025-09-07T08:04:52.7319930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7320268Z     return self._forward_impl(x)
2025-09-07T08:04:52.7320614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7320955Z     x = self.layer4(x)
2025-09-07T08:04:52.7321251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T08:04:52.7321582Z     out = self.relu(out)
2025-09-07T08:04:52.7321674Z 
2025-09-07T08:04:52.7321772Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7322100Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7322407Z     return mod(*inputs)
2025-09-07T08:04:52.7322703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7323033Z     return self._forward_impl(x)
2025-09-07T08:04:52.7323356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T08:04:52.7323703Z     x = self.layer4(x)
2025-09-07T08:04:52.7323993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T08:04:52.7324321Z     out = self.conv2(out)
2025-09-07T08:04:52.7324418Z 
2025-09-07T08:04:52.7324515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7324844Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7325148Z     return mod(*inputs)
2025-09-07T08:04:52.7325441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7325773Z     return self._forward_impl(x)
2025-09-07T08:04:52.7326096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 278, in _forward_impl
2025-09-07T08:04:52.7326438Z     x = self.avgpool(x)
2025-09-07T08:04:52.7326535Z 
2025-09-07T08:04:52.7326624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:04:52.7328960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:04:52.7329273Z     return mod(*inputs)
2025-09-07T08:04:52.7329569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T08:04:52.7329907Z     return self._forward_impl(x)
2025-09-07T08:04:52.7330239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 280, in _forward_impl
2025-09-07T08:04:52.7330588Z     x = self.fc(x)
2025-09-07T08:04:52.7330675Z 
2025-09-07T08:05:07.3918865Z pass
2025-09-07T08:05:07.3921368Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:09.3984271Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:05:09.3985165Z   import pynvml  # type: ignore[import]
2025-09-07T08:05:11.4081639Z 
2025-09-07T08:05:11.5978676Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:05:11.5978967Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:05:11.5982875Z cpu  eval  functorch_maml_omniglot            
2025-09-07T08:05:11.6079147Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:11.6157682Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:11.6205173Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:13.8168273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8168723Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8169112Z     return mod(*inputs)
2025-09-07T08:05:13.8169221Z 
2025-09-07T08:05:13.8169313Z cudagraph partition due to non gpu ops
2025-09-07T08:05:13.8169531Z cudagraph partition due to non gpu ops
2025-09-07T08:05:13.8169759Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8170124Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8170456Z     return mod(*inputs)
2025-09-07T08:05:13.8170551Z 
2025-09-07T08:05:13.8170645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8171002Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8171320Z     return mod(*inputs)
2025-09-07T08:05:13.8171416Z 
2025-09-07T08:05:13.8171520Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8171866Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8172182Z     return mod(*inputs)
2025-09-07T08:05:13.8172290Z 
2025-09-07T08:05:13.8172389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8172737Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8173048Z     return mod(*inputs)
2025-09-07T08:05:13.8173139Z 
2025-09-07T08:05:13.8173231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8173579Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8173896Z     return mod(*inputs)
2025-09-07T08:05:13.8173991Z 
2025-09-07T08:05:13.8174109Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8174450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8174755Z     return mod(*inputs)
2025-09-07T08:05:13.8174864Z 
2025-09-07T08:05:13.8174961Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8175701Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8176020Z     return mod(*inputs)
2025-09-07T08:05:13.8176116Z 
2025-09-07T08:05:13.8176215Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8176560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8176874Z     return mod(*inputs)
2025-09-07T08:05:13.8176965Z 
2025-09-07T08:05:13.8177066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:13.8177403Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:05:13.8177772Z     return mod(*inputs)
2025-09-07T08:05:13.8177868Z 
2025-09-07T08:05:19.9970704Z pass
2025-09-07T08:05:19.9973758Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:21.9517851Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:05:21.9518862Z   import pynvml  # type: ignore[import]
2025-09-07T08:05:23.9561892Z 
2025-09-07T08:05:25.6349066Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:05:25.6349351Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:05:25.6400656Z cpu  eval  hf_Albert                          
2025-09-07T08:05:26.5545186Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:26.7960599Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:27.0941767Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:34.1765325Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1765801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1766176Z     return mod(**inputs)
2025-09-07T08:05:34.1766600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1767023Z     outputs = self.albert(
2025-09-07T08:05:34.1767414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 700, in forward
2025-09-07T08:05:34.1767819Z     embedding_output = self.embeddings(
2025-09-07T08:05:34.1768220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 240, in forward
2025-09-07T08:05:34.1768632Z     embeddings = self.LayerNorm(embeddings)
2025-09-07T08:05:34.1768770Z 
2025-09-07T08:05:34.1768855Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1769051Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1769247Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1769460Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1769650Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1769833Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1770022Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1770214Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1770431Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1770620Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1770814Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1770998Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1771230Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1771596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1771925Z     return mod(**inputs)
2025-09-07T08:05:34.1772294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1773084Z     outputs = self.albert(
2025-09-07T08:05:34.1773455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1773851Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1774254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 520, in forward
2025-09-07T08:05:34.1774696Z     hidden_states = self.embedding_hidden_mapping_in(hidden_states)
2025-09-07T08:05:34.1774887Z 
2025-09-07T08:05:34.1774988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1775348Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1775676Z     return mod(**inputs)
2025-09-07T08:05:34.1776040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1776431Z     outputs = self.albert(
2025-09-07T08:05:34.1776951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1777353Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1777796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1778240Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1778676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1779204Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1779729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1780331Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1780851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.1781240Z     self.query(hidden_states)
2025-09-07T08:05:34.1781366Z 
2025-09-07T08:05:34.1781466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1781826Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1782143Z     return mod(**inputs)
2025-09-07T08:05:34.1782499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1782895Z     outputs = self.albert(
2025-09-07T08:05:34.1783261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1783683Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1784073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1784493Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1784931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1785439Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1785992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1786498Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1786993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.1787373Z     self.key(hidden_states)
2025-09-07T08:05:34.1787577Z 
2025-09-07T08:05:34.1787678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1788070Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1788382Z     return mod(**inputs)
2025-09-07T08:05:34.1788732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1789131Z     outputs = self.albert(
2025-09-07T08:05:34.1789495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1789898Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1790276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1790714Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1791264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1791804Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1792334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1792843Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1793353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.1793757Z     self.value(hidden_states)
2025-09-07T08:05:34.1793879Z 
2025-09-07T08:05:34.1793954Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1794190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1794546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1794867Z     return mod(**inputs)
2025-09-07T08:05:34.1795233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1795623Z     outputs = self.albert(
2025-09-07T08:05:34.1795985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1796367Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1796746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1797176Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1797604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1798120Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1798633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1799137Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1799640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.1800100Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.1800294Z 
2025-09-07T08:05:34.1800401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1800743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1801079Z     return mod(**inputs)
2025-09-07T08:05:34.1801432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1801902Z     outputs = self.albert(
2025-09-07T08:05:34.1802286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1802680Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1803220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1803669Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1804101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1804617Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1805127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1805735Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1806256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.1806680Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.1806845Z 
2025-09-07T08:05:34.1806953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1807303Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1807624Z     return mod(**inputs)
2025-09-07T08:05:34.1807997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1808390Z     outputs = self.albert(
2025-09-07T08:05:34.1808748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1809139Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1809525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1809955Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1810378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1810885Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1811399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1811799Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1812206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1812606Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1813007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.1813413Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.1824776Z 
2025-09-07T08:05:34.1824929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1825341Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1825691Z     return mod(**inputs)
2025-09-07T08:05:34.1826108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1826524Z     outputs = self.albert(
2025-09-07T08:05:34.1826905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1827303Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1827740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1828343Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1828788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1829315Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1829840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1830245Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1830648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1831072Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1831553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.1831986Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.1832348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.1832805Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.1833052Z 
2025-09-07T08:05:34.1833157Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1833529Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1833856Z     return mod(**inputs)
2025-09-07T08:05:34.1834231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1834620Z     outputs = self.albert(
2025-09-07T08:05:34.1834987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1835385Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1835770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1836207Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1836623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1837147Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1837665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1838073Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1838470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1838880Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1839273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.1839679Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.1839813Z 
2025-09-07T08:05:34.1839921Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1840270Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1840591Z     return mod(**inputs)
2025-09-07T08:05:34.1840953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1841338Z     outputs = self.albert(
2025-09-07T08:05:34.1841702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1842082Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1842545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1842971Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1843391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1843905Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1844409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1844917Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1845430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.1845825Z     self.query(hidden_states)
2025-09-07T08:05:34.1846007Z 
2025-09-07T08:05:34.1846125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1846475Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1846792Z     return mod(**inputs)
2025-09-07T08:05:34.1847154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1847542Z     outputs = self.albert(
2025-09-07T08:05:34.1847904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1848296Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1848674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1849096Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1849527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1850035Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1850553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1851064Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1851571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.1851956Z     self.key(hidden_states)
2025-09-07T08:05:34.1852064Z 
2025-09-07T08:05:34.1852161Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1852508Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1852827Z     return mod(**inputs)
2025-09-07T08:05:34.1853195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1853584Z     outputs = self.albert(
2025-09-07T08:05:34.1853944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1854376Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1854742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1855168Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1855585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1856099Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1856710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1857218Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1857779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.1858180Z     self.value(hidden_states)
2025-09-07T08:05:34.1858305Z 
2025-09-07T08:05:34.1858384Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1858622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1858977Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1859302Z     return mod(**inputs)
2025-09-07T08:05:34.1859668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1860064Z     outputs = self.albert(
2025-09-07T08:05:34.1860525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1860915Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1861301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1861728Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1862153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1862667Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1863178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1863690Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1864349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.1864814Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.1865009Z 
2025-09-07T08:05:34.1865115Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1865467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1865783Z     return mod(**inputs)
2025-09-07T08:05:34.1866138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1866519Z     outputs = self.albert(
2025-09-07T08:05:34.1866882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1867271Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1867652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1868087Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1868511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1869014Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1869526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1870028Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1870526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.1871032Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.1871196Z 
2025-09-07T08:05:34.1871300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1871641Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1871954Z     return mod(**inputs)
2025-09-07T08:05:34.1872309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1872693Z     outputs = self.albert(
2025-09-07T08:05:34.1873044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1873434Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1873812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1874232Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1874724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1875232Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1875747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1876142Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1876543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1876938Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1877326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.1877724Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.1877861Z 
2025-09-07T08:05:34.1877962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1878314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1878623Z     return mod(**inputs)
2025-09-07T08:05:34.1878978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1879363Z     outputs = self.albert(
2025-09-07T08:05:34.1879720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1880108Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1880478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1880899Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1881321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1881833Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1882342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1882734Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1883124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1883521Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1883910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.1884348Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.1884699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.1886170Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.1886414Z 
2025-09-07T08:05:34.1886510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1886861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1887176Z     return mod(**inputs)
2025-09-07T08:05:34.1887535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1887916Z     outputs = self.albert(
2025-09-07T08:05:34.1888278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1888666Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1889043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1889546Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1889968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1890480Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1890989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1891382Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1891778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1892164Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1892553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.1892959Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.1893091Z 
2025-09-07T08:05:34.1893190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1893537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1893842Z     return mod(**inputs)
2025-09-07T08:05:34.1894195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1894573Z     outputs = self.albert(
2025-09-07T08:05:34.1894931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1895312Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1895684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1896105Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1896533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1897041Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1897615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1898124Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1898630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.1899018Z     self.query(hidden_states)
2025-09-07T08:05:34.1899124Z 
2025-09-07T08:05:34.1899225Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1899564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1899959Z     return mod(**inputs)
2025-09-07T08:05:34.1900317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1900704Z     outputs = self.albert(
2025-09-07T08:05:34.1901066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1901449Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1901829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1902246Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1902660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1903329Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1903961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1904465Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1904972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.1905360Z     self.key(hidden_states)
2025-09-07T08:05:34.1905468Z 
2025-09-07T08:05:34.1905564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1905912Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1906225Z     return mod(**inputs)
2025-09-07T08:05:34.1906585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1906971Z     outputs = self.albert(
2025-09-07T08:05:34.1907322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1907706Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1908079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1908495Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1908912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1909414Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1909918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1910418Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1910925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.1911309Z     self.value(hidden_states)
2025-09-07T08:05:34.1911414Z 
2025-09-07T08:05:34.1911489Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1911713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1912051Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1912350Z     return mod(**inputs)
2025-09-07T08:05:34.1912689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1913059Z     outputs = self.albert(
2025-09-07T08:05:34.1913406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1913919Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1914295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1914710Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1915124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1915630Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1916133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1916624Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1917114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.1917643Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.1917842Z 
2025-09-07T08:05:34.1917939Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1918287Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1918590Z     return mod(**inputs)
2025-09-07T08:05:34.1918936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1919311Z     outputs = self.albert(
2025-09-07T08:05:34.1919661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1920045Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1920412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1920827Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1921243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1921743Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1922241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1922743Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1923235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.1923653Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.1923845Z 
2025-09-07T08:05:34.1923953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1924305Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1924614Z     return mod(**inputs)
2025-09-07T08:05:34.1924966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1925344Z     outputs = self.albert(
2025-09-07T08:05:34.1925691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1926068Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1926431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1926842Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1927248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1927753Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1928336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1928728Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1929118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1929514Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1929895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.1930285Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.1930411Z 
2025-09-07T08:05:34.1930503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1930837Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1931141Z     return mod(**inputs)
2025-09-07T08:05:34.1931555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1931930Z     outputs = self.albert(
2025-09-07T08:05:34.1932280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1932660Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1933033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1933443Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1933854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1934361Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1934871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1935258Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1935666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1936049Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1936427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.1936825Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.1937167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.1937667Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.1937901Z 
2025-09-07T08:05:34.1938004Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1938351Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1938740Z     return mod(**inputs)
2025-09-07T08:05:34.1939103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1939480Z     outputs = self.albert(
2025-09-07T08:05:34.1939828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1940202Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1940565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1940978Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1941406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1942045Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1942565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1942952Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1943350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1943753Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1944136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.1944549Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.1944686Z 
2025-09-07T08:05:34.1944786Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1945198Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1945531Z     return mod(**inputs)
2025-09-07T08:05:34.1945891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1946292Z     outputs = self.albert(
2025-09-07T08:05:34.1946686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1947089Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1947479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1947899Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1948314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1948842Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1949363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1949868Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1950365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.1950737Z     self.query(hidden_states)
2025-09-07T08:05:34.1950858Z 
2025-09-07T08:05:34.1950955Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1951291Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1951608Z     return mod(**inputs)
2025-09-07T08:05:34.1951952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1952351Z     outputs = self.albert(
2025-09-07T08:05:34.1952706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1953105Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1953472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1953901Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1954311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1954820Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1955316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1955897Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1956388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.1956775Z     self.key(hidden_states)
2025-09-07T08:05:34.1956879Z 
2025-09-07T08:05:34.1956985Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1957316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1957618Z     return mod(**inputs)
2025-09-07T08:05:34.1957970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1958351Z     outputs = self.albert(
2025-09-07T08:05:34.1958699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1959085Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1959532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1959979Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1960406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1960914Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1961429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1961928Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1962427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.1962824Z     self.value(hidden_states)
2025-09-07T08:05:34.1962934Z 
2025-09-07T08:05:34.1963012Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.1963242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1963590Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1963908Z     return mod(**inputs)
2025-09-07T08:05:34.1964257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1964639Z     outputs = self.albert(
2025-09-07T08:05:34.1965000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1965386Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1965759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1966170Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1966594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1967105Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1967615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1968118Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1968611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.1969069Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.1969271Z 
2025-09-07T08:05:34.1969364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1969815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1970132Z     return mod(**inputs)
2025-09-07T08:05:34.1970477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1970859Z     outputs = self.albert(
2025-09-07T08:05:34.1971220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1971605Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1971969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1972389Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1972806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1973387Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1973899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.1974400Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.1974898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.1975321Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.1975490Z 
2025-09-07T08:05:34.1975585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1975933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1976241Z     return mod(**inputs)
2025-09-07T08:05:34.1976599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1976987Z     outputs = self.albert(
2025-09-07T08:05:34.1977349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1977782Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1978156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1978580Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1979001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1979513Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1980021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1980421Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1980825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1981222Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1981612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.1982011Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.1982138Z 
2025-09-07T08:05:34.1982237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1982587Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1982901Z     return mod(**inputs)
2025-09-07T08:05:34.1983260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1983718Z     outputs = self.albert(
2025-09-07T08:05:34.1984081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1984470Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1984843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1985261Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1985675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1986179Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1986689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1987083Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1987552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1987943Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1988330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.1988736Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.1989092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.1989547Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.1989782Z 
2025-09-07T08:05:34.1989878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1990231Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1990545Z     return mod(**inputs)
2025-09-07T08:05:34.1990908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1991294Z     outputs = self.albert(
2025-09-07T08:05:34.1991654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1992039Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1992412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1992833Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1993243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.1993751Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.1994262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.1994660Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.1995053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.1995438Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.1995819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.1996216Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.1996346Z 
2025-09-07T08:05:34.1996445Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.1996789Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.1997098Z     return mod(**inputs)
2025-09-07T08:05:34.1997452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.1997931Z     outputs = self.albert(
2025-09-07T08:05:34.1998297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.1998674Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.1999050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.1999473Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.1999892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2000403Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2000903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2001486Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2001992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2002377Z     self.query(hidden_states)
2025-09-07T08:05:34.2002484Z 
2025-09-07T08:05:34.2002587Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2002924Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2003361Z     return mod(**inputs)
2025-09-07T08:05:34.2003718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2004097Z     outputs = self.albert(
2025-09-07T08:05:34.2004456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2004845Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2005221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2005639Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2006060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2006564Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2007064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2007564Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2008072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2008461Z     self.key(hidden_states)
2025-09-07T08:05:34.2008568Z 
2025-09-07T08:05:34.2008668Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2009003Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2009312Z     return mod(**inputs)
2025-09-07T08:05:34.2009668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2010049Z     outputs = self.albert(
2025-09-07T08:05:34.2010404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2010789Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2011161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2011584Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2012134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2012632Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2013138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2013636Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2014135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2014517Z     self.value(hidden_states)
2025-09-07T08:05:34.2014621Z 
2025-09-07T08:05:34.2014699Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2014929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2015400Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2015727Z     return mod(**inputs)
2025-09-07T08:05:34.2016080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2016468Z     outputs = self.albert(
2025-09-07T08:05:34.2016830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2017221Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2017653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2018078Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2018506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2019033Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2019548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2020050Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2020543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2020998Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2021200Z 
2025-09-07T08:05:34.2021298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2021650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2021966Z     return mod(**inputs)
2025-09-07T08:05:34.2022319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2022705Z     outputs = self.albert(
2025-09-07T08:05:34.2023065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2023449Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2023819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2024239Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2024662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2025170Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2025680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2026272Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2026770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2027198Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2027366Z 
2025-09-07T08:05:34.2027460Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2027805Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2028112Z     return mod(**inputs)
2025-09-07T08:05:34.2028466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2028844Z     outputs = self.albert(
2025-09-07T08:05:34.2029205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2029713Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2030081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2030503Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2030921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2031429Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2031945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2032332Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2032728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2033128Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2033519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2033917Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2034045Z 
2025-09-07T08:05:34.2034139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2034488Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2034805Z     return mod(**inputs)
2025-09-07T08:05:34.2035169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2035555Z     outputs = self.albert(
2025-09-07T08:05:34.2035927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2036316Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2036697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2037121Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2037533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2038044Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2038557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2038952Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2039346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2039732Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2040234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2040641Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2040993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2041446Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2041681Z 
2025-09-07T08:05:34.2041780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2042137Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2042459Z     return mod(**inputs)
2025-09-07T08:05:34.2042816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2043201Z     outputs = self.albert(
2025-09-07T08:05:34.2043650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2044049Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2044429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2044853Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2045265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2045786Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2046295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2046695Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2047100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2047487Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2047878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2048279Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2048411Z 
2025-09-07T08:05:34.2048513Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2048867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2049177Z     return mod(**inputs)
2025-09-07T08:05:34.2049535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2049921Z     outputs = self.albert(
2025-09-07T08:05:34.2050282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2050667Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2051046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2051467Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2051889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2052397Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2052909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2053426Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2053943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2054422Z     self.query(hidden_states)
2025-09-07T08:05:34.2054538Z 
2025-09-07T08:05:34.2054643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2054990Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2055306Z     return mod(**inputs)
2025-09-07T08:05:34.2055668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2056054Z     outputs = self.albert(
2025-09-07T08:05:34.2056414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2056801Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2057179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2057975Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2058411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2058930Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2059441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2059946Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2060451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2060837Z     self.key(hidden_states)
2025-09-07T08:05:34.2060945Z 
2025-09-07T08:05:34.2061050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2061399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2061718Z     return mod(**inputs)
2025-09-07T08:05:34.2062079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2062467Z     outputs = self.albert(
2025-09-07T08:05:34.2062826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2063216Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2063595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2064023Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2064448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2064965Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2065483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2065985Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2066498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2066884Z     self.value(hidden_states)
2025-09-07T08:05:34.2066991Z 
2025-09-07T08:05:34.2067068Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2067302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2067650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2067961Z     return mod(**inputs)
2025-09-07T08:05:34.2068311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2068780Z     outputs = self.albert(
2025-09-07T08:05:34.2069138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2069527Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2069904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2070319Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2070738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2071245Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2071757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2072357Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2072856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2073315Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2073519Z 
2025-09-07T08:05:34.2073615Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2073969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2074285Z     return mod(**inputs)
2025-09-07T08:05:34.2074634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2075019Z     outputs = self.albert(
2025-09-07T08:05:34.2075387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2075784Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2076160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2076580Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2077003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2077513Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2078023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2078529Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2079029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2079458Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2079628Z 
2025-09-07T08:05:34.2079731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2080088Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2080405Z     return mod(**inputs)
2025-09-07T08:05:34.2080755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2081133Z     outputs = self.albert(
2025-09-07T08:05:34.2081491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2081877Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2082243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2082754Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2083180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2083697Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2084204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2084597Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2085007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2085403Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2085808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2086294Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2086430Z 
2025-09-07T08:05:34.2086535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2086896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2087222Z     return mod(**inputs)
2025-09-07T08:05:34.2087588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2087974Z     outputs = self.albert(
2025-09-07T08:05:34.2088343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2088734Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2089119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2089548Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2089974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2090496Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2091020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2091423Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2091824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2092215Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2092606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2093013Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2093370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2093829Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2094061Z 
2025-09-07T08:05:34.2094164Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2094520Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2094839Z     return mod(**inputs)
2025-09-07T08:05:34.2095202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2095587Z     outputs = self.albert(
2025-09-07T08:05:34.2095940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2096329Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2096717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2097232Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2097684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2098198Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2098713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2099110Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2099505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2099892Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2100374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2100790Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2100922Z 
2025-09-07T08:05:34.2101025Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2101372Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2101681Z     return mod(**inputs)
2025-09-07T08:05:34.2102038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2102419Z     outputs = self.albert(
2025-09-07T08:05:34.2102779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2103284Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2103658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2104097Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2104518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2105028Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2105530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2106048Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2106545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2106932Z     self.query(hidden_states)
2025-09-07T08:05:34.2107042Z 
2025-09-07T08:05:34.2107143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2107485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2107792Z     return mod(**inputs)
2025-09-07T08:05:34.2108142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2108517Z     outputs = self.albert(
2025-09-07T08:05:34.2108872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2109252Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2109621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2110036Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2110451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2111151Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2111653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2112151Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2112647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2113028Z     self.key(hidden_states)
2025-09-07T08:05:34.2113136Z 
2025-09-07T08:05:34.2113235Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2113572Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2113881Z     return mod(**inputs)
2025-09-07T08:05:34.2114229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2114709Z     outputs = self.albert(
2025-09-07T08:05:34.2115058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2115443Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2115813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2116232Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2116646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2117149Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2117647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2118146Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2118649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2119036Z     self.value(hidden_states)
2025-09-07T08:05:34.2119140Z 
2025-09-07T08:05:34.2119222Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2119808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2120264Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2129756Z     return mod(**inputs)
2025-09-07T08:05:34.2130208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2130639Z     outputs = self.albert(
2025-09-07T08:05:34.2131038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2131448Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2131849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2132284Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2132722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2133248Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2133769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2134276Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2134793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2135402Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2135605Z 
2025-09-07T08:05:34.2135718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2136086Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2136410Z     return mod(**inputs)
2025-09-07T08:05:34.2136769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2137159Z     outputs = self.albert(
2025-09-07T08:05:34.2137598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2137986Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2138363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2138890Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2139320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2139838Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2140355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2140863Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2141363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2141790Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2141965Z 
2025-09-07T08:05:34.2142066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2142432Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2142746Z     return mod(**inputs)
2025-09-07T08:05:34.2143107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2143495Z     outputs = self.albert(
2025-09-07T08:05:34.2143855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2144247Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2144615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2145038Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2145460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2145980Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2146488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2146890Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2147298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2147698Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2148093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2148497Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2148625Z 
2025-09-07T08:05:34.2148726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2149085Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2149508Z     return mod(**inputs)
2025-09-07T08:05:34.2149869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2150250Z     outputs = self.albert(
2025-09-07T08:05:34.2150615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2151005Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2151394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2151823Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2152243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2152845Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2153367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2153763Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2154163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2154541Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2154916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2155319Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2155659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2156110Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2156344Z 
2025-09-07T08:05:34.2156443Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2156787Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2157089Z     return mod(**inputs)
2025-09-07T08:05:34.2157438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2157812Z     outputs = self.albert(
2025-09-07T08:05:34.2158160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2158539Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2158907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2159318Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2159734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2160234Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2160736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2161118Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2161498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2161881Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2162256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2162649Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2162778Z 
2025-09-07T08:05:34.2162875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2163303Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2163609Z     return mod(**inputs)
2025-09-07T08:05:34.2163955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2164332Z     outputs = self.albert(
2025-09-07T08:05:34.2164685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2165068Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2165441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2165857Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2166269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2166839Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2167345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2167852Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2168349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2168724Z     self.query(hidden_states)
2025-09-07T08:05:34.2168831Z 
2025-09-07T08:05:34.2168932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2169272Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2169576Z     return mod(**inputs)
2025-09-07T08:05:34.2169937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2170312Z     outputs = self.albert(
2025-09-07T08:05:34.2170664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2171044Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2171413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2171827Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2172239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2172735Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2173235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2173735Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2174228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2174607Z     self.key(hidden_states)
2025-09-07T08:05:34.2174710Z 
2025-09-07T08:05:34.2174803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2175145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2175457Z     return mod(**inputs)
2025-09-07T08:05:34.2175811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2176177Z     outputs = self.albert(
2025-09-07T08:05:34.2176533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2177001Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2177376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2177838Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2178248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2178748Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2179257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2179756Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2180251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2180712Z     self.value(hidden_states)
2025-09-07T08:05:34.2180823Z 
2025-09-07T08:05:34.2180899Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2181125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2181466Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2181774Z     return mod(**inputs)
2025-09-07T08:05:34.2182116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2182507Z     outputs = self.albert(
2025-09-07T08:05:34.2182861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2183241Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2183601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2184024Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2184435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2184934Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2185434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2185923Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2186419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2186871Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2187069Z 
2025-09-07T08:05:34.2187168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2187506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2187811Z     return mod(**inputs)
2025-09-07T08:05:34.2188169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2188540Z     outputs = self.albert(
2025-09-07T08:05:34.2188891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2189269Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2189639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2190056Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2190465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2191062Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2191560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2192055Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2192544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2192954Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2193117Z 
2025-09-07T08:05:34.2193209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2193545Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2193853Z     return mod(**inputs)
2025-09-07T08:05:34.2194263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2194644Z     outputs = self.albert(
2025-09-07T08:05:34.2194998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2195378Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2195751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2196160Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2196571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2197076Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2197581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2197972Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2198360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2198748Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2199129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2199203Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2199207Z 
2025-09-07T08:05:34.2199304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2199490Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2199551Z     return mod(**inputs)
2025-09-07T08:05:34.2199807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2199874Z     outputs = self.albert(
2025-09-07T08:05:34.2200128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2200191Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2200446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2200543Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2200800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2200982Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2201232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2201438Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2201694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2201770Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2202025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2202102Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2202317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2202493Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2202497Z 
2025-09-07T08:05:34.2202593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2202781Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2202919Z     return mod(**inputs)
2025-09-07T08:05:34.2203334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2203392Z     outputs = self.albert(
2025-09-07T08:05:34.2203652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2203717Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2203971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2204074Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2204327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2204516Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2204776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2204855Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2205111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2205178Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2205434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2205506Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2205510Z 
2025-09-07T08:05:34.2205602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2205788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2205850Z     return mod(**inputs)
2025-09-07T08:05:34.2206109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2206167Z     outputs = self.albert(
2025-09-07T08:05:34.2206425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2206488Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2206739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2206835Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2207087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2207269Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2207522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2207882Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2208137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2208205Z     self.query(hidden_states)
2025-09-07T08:05:34.2208209Z 
2025-09-07T08:05:34.2208303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2208487Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2208544Z     return mod(**inputs)
2025-09-07T08:05:34.2208793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2208855Z     outputs = self.albert(
2025-09-07T08:05:34.2209214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2209283Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2209533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2209632Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2209883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2210066Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2210317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2210492Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2210745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2210812Z     self.key(hidden_states)
2025-09-07T08:05:34.2210815Z 
2025-09-07T08:05:34.2210906Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2211091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2211147Z     return mod(**inputs)
2025-09-07T08:05:34.2211398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2211456Z     outputs = self.albert(
2025-09-07T08:05:34.2211705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2211771Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2212019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2212123Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2212371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2212553Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2212805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2212978Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2213231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2213296Z     self.value(hidden_states)
2025-09-07T08:05:34.2213300Z 
2025-09-07T08:05:34.2213377Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2213544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2213728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2213790Z     return mod(**inputs)
2025-09-07T08:05:34.2214043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2214106Z     outputs = self.albert(
2025-09-07T08:05:34.2214356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2214420Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2214672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2214767Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2215022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2215279Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2215539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2215712Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2215963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2216103Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2216107Z 
2025-09-07T08:05:34.2216196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2216380Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2216436Z     return mod(**inputs)
2025-09-07T08:05:34.2216695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2216753Z     outputs = self.albert(
2025-09-07T08:05:34.2217002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2217065Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2217313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2217446Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2217697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2217880Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2218135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2218310Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2218562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2218663Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2218666Z 
2025-09-07T08:05:34.2218757Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2218939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2218995Z     return mod(**inputs)
2025-09-07T08:05:34.2219246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2219303Z     outputs = self.albert(
2025-09-07T08:05:34.2219559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2219700Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2219950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2220046Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2220296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2220478Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2220726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2220799Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2221396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2221469Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2221728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2221801Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2221804Z 
2025-09-07T08:05:34.2221896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2222079Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2222139Z     return mod(**inputs)
2025-09-07T08:05:34.2222392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2222450Z     outputs = self.albert(
2025-09-07T08:05:34.2222710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2222780Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2223036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2223133Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2223383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2223567Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2223819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2223891Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2224144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2224220Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2224474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2224548Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2224760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2224930Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2224933Z 
2025-09-07T08:05:34.2225025Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2225207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2225263Z     return mod(**inputs)
2025-09-07T08:05:34.2225515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2225680Z     outputs = self.albert(
2025-09-07T08:05:34.2225934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2225999Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2226256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2226357Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2226610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2226796Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2227045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2227116Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2227438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2227504Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2227760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2227830Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2227833Z 
2025-09-07T08:05:34.2227923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2228103Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2228161Z     return mod(**inputs)
2025-09-07T08:05:34.2228411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2228468Z     outputs = self.albert(
2025-09-07T08:05:34.2228853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2228920Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2229175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2229272Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2229521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2229703Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2229951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2230131Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2230386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2230450Z     self.query(hidden_states)
2025-09-07T08:05:34.2230453Z 
2025-09-07T08:05:34.2230540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2230723Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2230785Z     return mod(**inputs)
2025-09-07T08:05:34.2231037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2231103Z     outputs = self.albert(
2025-09-07T08:05:34.2231355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2231419Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2231681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2231853Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2232111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2232290Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2232547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2232723Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2232974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2233040Z     self.key(hidden_states)
2025-09-07T08:05:34.2233043Z 
2025-09-07T08:05:34.2233133Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2233390Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2233451Z     return mod(**inputs)
2025-09-07T08:05:34.2233707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2233768Z     outputs = self.albert(
2025-09-07T08:05:34.2234017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2234087Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2234336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2234443Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2234694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2234886Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2235143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2235317Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2235576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2235636Z     self.value(hidden_states)
2025-09-07T08:05:34.2235639Z 
2025-09-07T08:05:34.2235712Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2235800Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2235982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2236041Z     return mod(**inputs)
2025-09-07T08:05:34.2236296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2236355Z     outputs = self.albert(
2025-09-07T08:05:34.2236603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2236664Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2236916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2237013Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2237265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2237444Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2237699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2237949Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2238199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2238334Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2238337Z 
2025-09-07T08:05:34.2238425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2238609Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2238665Z     return mod(**inputs)
2025-09-07T08:05:34.2238916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2238976Z     outputs = self.albert(
2025-09-07T08:05:34.2239294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2239359Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2239610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2239707Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2239958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2240139Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2240390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2240562Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2240818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2240918Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2240922Z 
2025-09-07T08:05:34.2241010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2241192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2241247Z     return mod(**inputs)
2025-09-07T08:05:34.2241499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2241555Z     outputs = self.albert(
2025-09-07T08:05:34.2241805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2241866Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2242117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2242218Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2242466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2242647Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2242897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2242969Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2243224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2243290Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2243550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2243708Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2243711Z 
2025-09-07T08:05:34.2243800Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2243979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2244036Z     return mod(**inputs)
2025-09-07T08:05:34.2244287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2244343Z     outputs = self.albert(
2025-09-07T08:05:34.2244595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2244657Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2244906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2245106Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2245359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2245544Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2245797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2245871Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2246125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2246193Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2246454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2246534Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2246751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2246923Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2246926Z 
2025-09-07T08:05:34.2247018Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2247202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2247260Z     return mod(**inputs)
2025-09-07T08:05:34.2247513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2247571Z     outputs = self.albert(
2025-09-07T08:05:34.2247827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2247895Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2248148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2248252Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2248505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2248691Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2248942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2249017Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2249272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2249337Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2249684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2249757Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2249760Z 
2025-09-07T08:05:34.2249856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2250044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2250104Z     return mod(**inputs)
2025-09-07T08:05:34.2250358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2250418Z     outputs = self.albert(
2025-09-07T08:05:34.2250674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2250737Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2251059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2251168Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2251421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2251614Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2251865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2252049Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2252301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2252363Z     self.query(hidden_states)
2025-09-07T08:05:34.2252372Z 
2025-09-07T08:05:34.2252469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2252650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2252707Z     return mod(**inputs)
2025-09-07T08:05:34.2252957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2253017Z     outputs = self.albert(
2025-09-07T08:05:34.2253267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2253330Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2253583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2253677Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2253929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2254111Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2254360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2254536Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2254784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2254846Z     self.key(hidden_states)
2025-09-07T08:05:34.2254849Z 
2025-09-07T08:05:34.2254938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2255121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2255177Z     return mod(**inputs)
2025-09-07T08:05:34.2255516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2255576Z     outputs = self.albert(
2025-09-07T08:05:34.2255825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2255889Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2256139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2256239Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2256492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2256673Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2256997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2257179Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2257477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2257542Z     self.value(hidden_states)
2025-09-07T08:05:34.2257545Z 
2025-09-07T08:05:34.2257616Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2257707Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2257891Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2257948Z     return mod(**inputs)
2025-09-07T08:05:34.2258198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2258256Z     outputs = self.albert(
2025-09-07T08:05:34.2258515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2258577Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2258828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2258924Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2259173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2259353Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2259601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2259776Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2260033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2260168Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2260171Z 
2025-09-07T08:05:34.2260258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2260443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2260499Z     return mod(**inputs)
2025-09-07T08:05:34.2260748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2260810Z     outputs = self.albert(
2025-09-07T08:05:34.2261060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2261125Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2261375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2261552Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2261804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2261986Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2262238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2262409Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2262662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2262760Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2262767Z 
2025-09-07T08:05:34.2262926Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2263116Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2263172Z     return mod(**inputs)
2025-09-07T08:05:34.2263426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2263482Z     outputs = self.albert(
2025-09-07T08:05:34.2263734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2263798Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2264046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2264144Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2264394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2264591Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2264840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2264912Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2265172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2265238Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2265497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2265568Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2265572Z 
2025-09-07T08:05:34.2265661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2265856Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2265912Z     return mod(**inputs)
2025-09-07T08:05:34.2266168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2266228Z     outputs = self.albert(
2025-09-07T08:05:34.2266485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2266549Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2266798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2266900Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2267151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2267419Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2267672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2267741Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2268002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2268070Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2268330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2268405Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2268614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2268874Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2268881Z 
2025-09-07T08:05:34.2268973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2269160Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2269219Z     return mod(**inputs)
2025-09-07T08:05:34.2269471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2269527Z     outputs = self.albert(
2025-09-07T08:05:34.2269777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2269858Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2270106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2270209Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2270461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2270647Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2270897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2270965Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2271222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2271290Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2271549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2271621Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2271626Z 
2025-09-07T08:05:34.2271716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2271902Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2271959Z     return mod(**inputs)
2025-09-07T08:05:34.2272222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2272282Z     outputs = self.albert(
2025-09-07T08:05:34.2272540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2272603Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2272853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2272955Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2273206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2273466Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2273716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2273891Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2274143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T08:05:34.2274208Z     self.query(hidden_states)
2025-09-07T08:05:34.2274212Z 
2025-09-07T08:05:34.2274303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2274489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2274554Z     return mod(**inputs)
2025-09-07T08:05:34.2274875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2274937Z     outputs = self.albert(
2025-09-07T08:05:34.2275192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2275253Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2275507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2275604Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2275853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2276036Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2276294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2276471Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2276720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T08:05:34.2276781Z     self.key(hidden_states)
2025-09-07T08:05:34.2276785Z 
2025-09-07T08:05:34.2276875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2277057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2277115Z     return mod(**inputs)
2025-09-07T08:05:34.2277364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2277425Z     outputs = self.albert(
2025-09-07T08:05:34.2277678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2277746Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2277997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2278092Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2278348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2278529Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2278784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2278955Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2279275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T08:05:34.2279341Z     self.value(hidden_states)
2025-09-07T08:05:34.2279344Z 
2025-09-07T08:05:34.2279416Z cudagraph partition due to non gpu ops
2025-09-07T08:05:34.2279510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2279691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2279750Z     return mod(**inputs)
2025-09-07T08:05:34.2279999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2280058Z     outputs = self.albert(
2025-09-07T08:05:34.2280310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2280371Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2280695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2280797Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2281045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2281228Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2281476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2281653Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2281904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T08:05:34.2282036Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:05:34.2282043Z 
2025-09-07T08:05:34.2282133Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2282317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2282380Z     return mod(**inputs)
2025-09-07T08:05:34.2282629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2282693Z     outputs = self.albert(
2025-09-07T08:05:34.2282940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2283002Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2283254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2283351Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2283608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2283788Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2284042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T08:05:34.2284213Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T08:05:34.2284463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T08:05:34.2284568Z     projected_context_layer = self.dense(attention_output)
2025-09-07T08:05:34.2284571Z 
2025-09-07T08:05:34.2284659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2284842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2284977Z     return mod(**inputs)
2025-09-07T08:05:34.2285235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2285294Z     outputs = self.albert(
2025-09-07T08:05:34.2285542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2285609Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2285858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2285959Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2286210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2286391Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2286717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2286788Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2287043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2287110Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2287371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T08:05:34.2287439Z     ffn_output = self.ffn(attention_output)
2025-09-07T08:05:34.2287442Z 
2025-09-07T08:05:34.2287530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2287717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2287773Z     return mod(**inputs)
2025-09-07T08:05:34.2288030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2288087Z     outputs = self.albert(
2025-09-07T08:05:34.2288337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2288405Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2288653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2288756Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2289005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2289184Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2289439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2289511Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2289764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2289828Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2290084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T08:05:34.2290159Z     ffn_output = self.activation(ffn_output)
2025-09-07T08:05:34.2290364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:05:34.2290537Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:05:34.2290540Z 
2025-09-07T08:05:34.2290627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2290916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2290974Z     return mod(**inputs)
2025-09-07T08:05:34.2291229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T08:05:34.2291290Z     outputs = self.albert(
2025-09-07T08:05:34.2291537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T08:05:34.2291606Z     encoder_outputs = self.encoder(
2025-09-07T08:05:34.2291856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T08:05:34.2291959Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T08:05:34.2292207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T08:05:34.2292457Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T08:05:34.2292721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T08:05:34.2292797Z     ffn_output = apply_chunking_to_forward(
2025-09-07T08:05:34.2293056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:05:34.2293122Z     return forward_fn(*input_tensors)
2025-09-07T08:05:34.2293380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T08:05:34.2293454Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T08:05:34.2293458Z 
2025-09-07T08:05:34.2293548Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2293733Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2293794Z     return mod(**inputs)
2025-09-07T08:05:34.2294049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 984, in forward
2025-09-07T08:05:34.2294155Z     prediction_scores = self.predictions(sequence_outputs)
2025-09-07T08:05:34.2294405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 866, in forward
2025-09-07T08:05:34.2294486Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:05:34.2294489Z 
2025-09-07T08:05:34.2294577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:05:34.2294761Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:05:34.2294818Z     return mod(**inputs)
2025-09-07T08:05:34.2295070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 984, in forward
2025-09-07T08:05:34.2295174Z     prediction_scores = self.predictions(sequence_outputs)
2025-09-07T08:05:34.2295426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 869, in forward
2025-09-07T08:05:34.2295511Z     hidden_states = self.decoder(hidden_states)
2025-09-07T08:05:34.2295515Z 
2025-09-07T08:05:44.0644991Z pass
2025-09-07T08:05:44.0645373Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:46.5245454Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:05:46.5246632Z   import pynvml  # type: ignore[import]
2025-09-07T08:05:48.5314102Z 
2025-09-07T08:05:52.7043802Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:05:52.7044138Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:05:52.7335890Z cpu  eval  hf_Bart                            
2025-09-07T08:05:54.0808191Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:54.4314342Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:05:54.7777835Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:03.9736327Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9736629Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9736852Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9737046Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9737255Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9737449Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9737722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9738531Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9738892Z     return mod(**inputs)
2025-09-07T08:06:03.9739307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9739724Z     outputs = self.model(
2025-09-07T08:06:03.9740122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9740527Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9740909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9741331Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9741680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9742049Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9742435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9742848Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9743258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:03.9743724Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:03.9743933Z 
2025-09-07T08:06:03.9744039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9744415Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9744738Z     return mod(**inputs)
2025-09-07T08:06:03.9745104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9745483Z     outputs = self.model(
2025-09-07T08:06:03.9745831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9746215Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9746587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9746969Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9747313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9747675Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9748055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9748454Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9748844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:03.9749218Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:03.9749503Z 
2025-09-07T08:06:03.9749681Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9750035Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9750362Z     return mod(**inputs)
2025-09-07T08:06:03.9750724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9751098Z     outputs = self.model(
2025-09-07T08:06:03.9751446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9751825Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9752208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9752587Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9752983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9753348Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9753728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9754132Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9754531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:03.9754957Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:03.9755098Z 
2025-09-07T08:06:03.9755175Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9755378Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9755570Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9755757Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9755977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9756336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9756657Z     return mod(**inputs)
2025-09-07T08:06:03.9757008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9757388Z     outputs = self.model(
2025-09-07T08:06:03.9757750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9758139Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9758527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9758897Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9759250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9759615Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9759999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9760388Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9760784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9761190Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9761639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:03.9762136Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:03.9762320Z 
2025-09-07T08:06:03.9762422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9762788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9763127Z     return mod(**inputs)
2025-09-07T08:06:03.9763574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9763991Z     outputs = self.model(
2025-09-07T08:06:03.9764352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9764740Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9765114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9765496Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9765841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9766208Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9766595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9767006Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9767475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9767888Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9768344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:03.9768834Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:03.9768994Z 
2025-09-07T08:06:03.9769102Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9769465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9769782Z     return mod(**inputs)
2025-09-07T08:06:03.9770142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9770525Z     outputs = self.model(
2025-09-07T08:06:03.9770886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9771275Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9771640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9772022Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9772362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9772728Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9773099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9773510Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9773904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:03.9774289Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:03.9774418Z 
2025-09-07T08:06:03.9774522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9774863Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9775185Z     return mod(**inputs)
2025-09-07T08:06:03.9775531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9775910Z     outputs = self.model(
2025-09-07T08:06:03.9776257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9776633Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9776992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9777365Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9777904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9778268Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9778669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9779100Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9779274Z 
2025-09-07T08:06:03.9779381Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9779732Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9780044Z     return mod(**inputs)
2025-09-07T08:06:03.9780394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9780779Z     outputs = self.model(
2025-09-07T08:06:03.9781196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9781595Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9781955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9782337Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9782677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9783033Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9783448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9783881Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9784259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:03.9784600Z     return self.act(input)
2025-09-07T08:06:03.9784706Z 
2025-09-07T08:06:03.9784813Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9785149Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9785468Z     return mod(**inputs)
2025-09-07T08:06:03.9785817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9786193Z     outputs = self.model(
2025-09-07T08:06:03.9786537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9786917Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9787278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9787660Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9788001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9788357Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9788728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T08:06:03.9789112Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:03.9789244Z 
2025-09-07T08:06:03.9789348Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9789687Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9790009Z     return mod(**inputs)
2025-09-07T08:06:03.9790369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9790750Z     outputs = self.model(
2025-09-07T08:06:03.9791098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9791594Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9791959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9792341Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9792686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9793051Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9793422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9793822Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9794211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:03.9794665Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:03.9794873Z 
2025-09-07T08:06:03.9795039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9795388Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9795705Z     return mod(**inputs)
2025-09-07T08:06:03.9796060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9796438Z     outputs = self.model(
2025-09-07T08:06:03.9796790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9797162Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9797529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9797904Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9798248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9798608Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9798981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9799373Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9799759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:03.9800141Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:03.9800267Z 
2025-09-07T08:06:03.9800363Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9800714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9801046Z     return mod(**inputs)
2025-09-07T08:06:03.9801401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9801796Z     outputs = self.model(
2025-09-07T08:06:03.9802158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9802541Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9802909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9803558Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9803957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9816206Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9816660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9817087Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9817622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:03.9818223Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:03.9818368Z 
2025-09-07T08:06:03.9818455Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9818665Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9818855Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9819046Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9819275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9819646Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9819973Z     return mod(**inputs)
2025-09-07T08:06:03.9820367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9820766Z     outputs = self.model(
2025-09-07T08:06:03.9821129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9821657Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9822035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9822422Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9822780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9823145Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9823519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9823924Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9824318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9824723Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9825182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:03.9825659Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:03.9825851Z 
2025-09-07T08:06:03.9825952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9826312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9826636Z     return mod(**inputs)
2025-09-07T08:06:03.9826993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9827366Z     outputs = self.model(
2025-09-07T08:06:03.9827717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9828100Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9828475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9828856Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9829210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9829572Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9829948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9830345Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9830725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9831118Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9831561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:03.9832103Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:03.9832262Z 
2025-09-07T08:06:03.9832363Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9832706Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9833022Z     return mod(**inputs)
2025-09-07T08:06:03.9833370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9833745Z     outputs = self.model(
2025-09-07T08:06:03.9834090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9834473Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9834836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9835215Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9835629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9835984Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9836386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9836795Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9837199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:03.9837595Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:03.9837730Z 
2025-09-07T08:06:03.9837833Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9838194Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9838522Z     return mod(**inputs)
2025-09-07T08:06:03.9838893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9839271Z     outputs = self.model(
2025-09-07T08:06:03.9839632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9840019Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9840395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9840779Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9841122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9841492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9841882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9842307Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9842486Z 
2025-09-07T08:06:03.9842592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9842933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9843250Z     return mod(**inputs)
2025-09-07T08:06:03.9843604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9843984Z     outputs = self.model(
2025-09-07T08:06:03.9844323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9844704Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9845069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9845445Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9845784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9846229Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9846605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9847019Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9847399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:03.9847736Z     return self.act(input)
2025-09-07T08:06:03.9847839Z 
2025-09-07T08:06:03.9847935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9848277Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9848591Z     return mod(**inputs)
2025-09-07T08:06:03.9848940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9850826Z     outputs = self.model(
2025-09-07T08:06:03.9851188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9851566Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9851929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9852304Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9852632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9852987Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9853358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T08:06:03.9853740Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:03.9853872Z 
2025-09-07T08:06:03.9853971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9854313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9854625Z     return mod(**inputs)
2025-09-07T08:06:03.9854971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9855346Z     outputs = self.model(
2025-09-07T08:06:03.9855688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9856060Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9856421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9856794Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9857124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9857477Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9857922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9858324Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9858715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:03.9859172Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:03.9859383Z 
2025-09-07T08:06:03.9859481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9859833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9860155Z     return mod(**inputs)
2025-09-07T08:06:03.9860509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9860975Z     outputs = self.model(
2025-09-07T08:06:03.9861330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9861709Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9862075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9862457Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9862790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9863145Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9863520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9863915Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9864294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:03.9864745Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:03.9864879Z 
2025-09-07T08:06:03.9864973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9865318Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9865634Z     return mod(**inputs)
2025-09-07T08:06:03.9865982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9866353Z     outputs = self.model(
2025-09-07T08:06:03.9866701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9867080Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9867445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9867820Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9868160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9868513Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9868885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9869268Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9869657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:03.9870048Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:03.9870193Z 
2025-09-07T08:06:03.9870268Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9870468Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9870665Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9870853Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9871076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9871425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9871736Z     return mod(**inputs)
2025-09-07T08:06:03.9872084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9872445Z     outputs = self.model(
2025-09-07T08:06:03.9872800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9873170Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9873526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9873888Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9874216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9874670Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9875042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9875424Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9875798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9876201Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9876689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:03.9877202Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:03.9877405Z 
2025-09-07T08:06:03.9877505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9877846Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9878256Z     return mod(**inputs)
2025-09-07T08:06:03.9878626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9878991Z     outputs = self.model(
2025-09-07T08:06:03.9879338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9879703Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9880057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9880421Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9880755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9881095Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9881461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9881851Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9882224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9882610Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9883035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:03.9883482Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:03.9883640Z 
2025-09-07T08:06:03.9883734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9884069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9884377Z     return mod(**inputs)
2025-09-07T08:06:03.9884718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9885084Z     outputs = self.model(
2025-09-07T08:06:03.9885428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9885796Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9886149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9886662Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9886993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9887337Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9887708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9888089Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9888552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:03.9888936Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:03.9889072Z 
2025-09-07T08:06:03.9889170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9889529Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9889835Z     return mod(**inputs)
2025-09-07T08:06:03.9890190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9890565Z     outputs = self.model(
2025-09-07T08:06:03.9890903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9891284Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9891708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9892092Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9892426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9892773Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9893137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9893559Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9893731Z 
2025-09-07T08:06:03.9893826Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9894162Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9894466Z     return mod(**inputs)
2025-09-07T08:06:03.9894805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9895178Z     outputs = self.model(
2025-09-07T08:06:03.9895520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9895886Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9896234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9896600Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9896924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9897269Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9897678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9898121Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9898512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:03.9898853Z     return self.act(input)
2025-09-07T08:06:03.9898959Z 
2025-09-07T08:06:03.9899056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9899397Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9899719Z     return mod(**inputs)
2025-09-07T08:06:03.9900080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9900454Z     outputs = self.model(
2025-09-07T08:06:03.9900790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9901168Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9901524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9901994Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9902336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9902694Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9903257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T08:06:03.9903640Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:03.9903770Z 
2025-09-07T08:06:03.9903875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9904224Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9904533Z     return mod(**inputs)
2025-09-07T08:06:03.9904888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9905262Z     outputs = self.model(
2025-09-07T08:06:03.9905735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9906113Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9906467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9906833Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9907170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9907534Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9907905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9908303Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9908687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:03.9909138Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:03.9909339Z 
2025-09-07T08:06:03.9909448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9909788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9910108Z     return mod(**inputs)
2025-09-07T08:06:03.9910446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9910814Z     outputs = self.model(
2025-09-07T08:06:03.9911158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9911525Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9911893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9912258Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9912594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9912947Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9913320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9913706Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9914084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:03.9914461Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:03.9914592Z 
2025-09-07T08:06:03.9914683Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9915018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9915329Z     return mod(**inputs)
2025-09-07T08:06:03.9915670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9916187Z     outputs = self.model(
2025-09-07T08:06:03.9916526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9916894Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9917249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9917620Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9917944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9918294Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9918655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9919045Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9919505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:03.9919884Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:03.9920018Z 
2025-09-07T08:06:03.9920090Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9920291Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9920487Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9920665Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9920880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9921216Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9921519Z     return mod(**inputs)
2025-09-07T08:06:03.9921871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9922239Z     outputs = self.model(
2025-09-07T08:06:03.9922580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9922947Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9923317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9923675Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9924010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9924351Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9924727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9925103Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9925480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9925876Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9926331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:03.9926813Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:03.9926992Z 
2025-09-07T08:06:03.9927086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9927425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9927738Z     return mod(**inputs)
2025-09-07T08:06:03.9928088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9928458Z     outputs = self.model(
2025-09-07T08:06:03.9928806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9929282Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9929643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9930017Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9930365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9930715Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9931096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9931481Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9931873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9932262Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9932805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:03.9933269Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:03.9933431Z 
2025-09-07T08:06:03.9933526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9933873Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9934183Z     return mod(**inputs)
2025-09-07T08:06:03.9934528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9934909Z     outputs = self.model(
2025-09-07T08:06:03.9935259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9935641Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9935993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9936373Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9936713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9937071Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9937435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9937872Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9938259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:03.9938644Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:03.9938773Z 
2025-09-07T08:06:03.9938876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9939210Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9939528Z     return mod(**inputs)
2025-09-07T08:06:03.9939881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9940254Z     outputs = self.model(
2025-09-07T08:06:03.9940598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9940973Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9941337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9941708Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9942044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9942390Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9942755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9943261Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9943430Z 
2025-09-07T08:06:03.9943529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9943869Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9944180Z     return mod(**inputs)
2025-09-07T08:06:03.9944526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9944894Z     outputs = self.model(
2025-09-07T08:06:03.9945243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9945618Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9945971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9946348Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9946757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9947113Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9947474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9947886Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9948265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:03.9948604Z     return self.act(input)
2025-09-07T08:06:03.9948708Z 
2025-09-07T08:06:03.9948810Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9949148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9949464Z     return mod(**inputs)
2025-09-07T08:06:03.9949817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9950195Z     outputs = self.model(
2025-09-07T08:06:03.9950535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9950911Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9951273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9951652Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9951994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9952341Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9952714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T08:06:03.9953095Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:03.9953232Z 
2025-09-07T08:06:03.9953335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9953673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9953982Z     return mod(**inputs)
2025-09-07T08:06:03.9954328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9954699Z     outputs = self.model(
2025-09-07T08:06:03.9955042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9955414Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9955779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9956153Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9956490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9956970Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9957354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9957755Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9958148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:03.9958601Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:03.9958807Z 
2025-09-07T08:06:03.9958913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9959257Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9959574Z     return mod(**inputs)
2025-09-07T08:06:03.9959990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9960380Z     outputs = self.model(
2025-09-07T08:06:03.9960728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9961109Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9961481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9961861Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9962207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9962560Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9962936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9963328Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9963723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:03.9964098Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:03.9964234Z 
2025-09-07T08:06:03.9964332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9964678Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9964995Z     return mod(**inputs)
2025-09-07T08:06:03.9965347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9965719Z     outputs = self.model(
2025-09-07T08:06:03.9966073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9966450Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9966815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9967192Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9967528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9967881Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9968255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9968652Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9969033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:03.9969415Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:03.9969555Z 
2025-09-07T08:06:03.9969630Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9969833Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9970029Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9970286Z cudagraph partition due to non gpu ops
2025-09-07T08:06:03.9970504Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9970851Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9971171Z     return mod(**inputs)
2025-09-07T08:06:03.9971514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9971889Z     outputs = self.model(
2025-09-07T08:06:03.9972240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9972622Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9972988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9973356Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9973776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9974140Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9974519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9974908Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9975304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9975705Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9976150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:03.9976633Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:03.9976818Z 
2025-09-07T08:06:03.9976916Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9977276Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9977660Z     return mod(**inputs)
2025-09-07T08:06:03.9978011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9978386Z     outputs = self.model(
2025-09-07T08:06:03.9978731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9979108Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9979473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9979853Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9980189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9980549Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9980937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9981332Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9981736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:03.9982128Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:03.9982577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:03.9983034Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:03.9983195Z 
2025-09-07T08:06:03.9983300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9983651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9984039Z     return mod(**inputs)
2025-09-07T08:06:03.9984404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9984783Z     outputs = self.model(
2025-09-07T08:06:03.9985135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9985511Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9985879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9986256Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9986595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9986952Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9987321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:03.9987789Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:03.9988181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:03.9988561Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:03.9988689Z 
2025-09-07T08:06:03.9988795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9989138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9989458Z     return mod(**inputs)
2025-09-07T08:06:03.9989806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9990179Z     outputs = self.model(
2025-09-07T08:06:03.9990521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9990898Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9991268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9991644Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9991977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9992342Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9992715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9993135Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9993302Z 
2025-09-07T08:06:03.9993403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9993737Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9994052Z     return mod(**inputs)
2025-09-07T08:06:03.9994403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:03.9994775Z     outputs = self.model(
2025-09-07T08:06:03.9995121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:03.9995490Z     encoder_outputs = self.encoder(
2025-09-07T08:06:03.9995855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:03.9996231Z     layer_outputs = encoder_layer(
2025-09-07T08:06:03.9996563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:03.9996914Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:03.9997281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:03.9997697Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:03.9998143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:03.9998485Z     return self.act(input)
2025-09-07T08:06:03.9998589Z 
2025-09-07T08:06:03.9998690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:03.9999050Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:03.9999366Z     return mod(**inputs)
2025-09-07T08:06:03.9999715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0000088Z     outputs = self.model(
2025-09-07T08:06:04.0000438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0000812Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0001240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0001630Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0001961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0002316Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0002689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T08:06:04.0003210Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0003344Z 
2025-09-07T08:06:04.0003448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0003828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0004159Z     return mod(**inputs)
2025-09-07T08:06:04.0004525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0004922Z     outputs = self.model(
2025-09-07T08:06:04.0005281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0005662Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0006041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0006423Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0006770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0007128Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0007512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:04.0007914Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:04.0008309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0008767Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0008968Z 
2025-09-07T08:06:04.0009069Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0009426Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0009745Z     return mod(**inputs)
2025-09-07T08:06:04.0010100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0010476Z     outputs = self.model(
2025-09-07T08:06:04.0010823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0011208Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0011581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0012121Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0012453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0012819Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0013199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:04.0013600Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:04.0013990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0014365Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0014508Z 
2025-09-07T08:06:04.0014606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0014958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0015381Z     return mod(**inputs)
2025-09-07T08:06:04.0015735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0016108Z     outputs = self.model(
2025-09-07T08:06:04.0016455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0016833Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0017201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0017612Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0017957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0018313Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0018695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:04.0019086Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:04.0019464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0019852Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0019998Z 
2025-09-07T08:06:04.0020078Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0020284Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0020473Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0020670Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0020890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0021235Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0021548Z     return mod(**inputs)
2025-09-07T08:06:04.0021892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0022271Z     outputs = self.model(
2025-09-07T08:06:04.0022619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0022999Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0023358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0023730Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0024065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0024420Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0024797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:04.0025182Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:04.0025652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0026059Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0026508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0026992Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0027173Z 
2025-09-07T08:06:04.0027270Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0027618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0027938Z     return mod(**inputs)
2025-09-07T08:06:04.0028287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0028656Z     outputs = self.model(
2025-09-07T08:06:04.0029096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0029484Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0029848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0030233Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0030563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0030916Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0031290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:04.0031679Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:04.0032060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0032457Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0032904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0033364Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0033524Z 
2025-09-07T08:06:04.0033629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0033972Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0034292Z     return mod(**inputs)
2025-09-07T08:06:04.0034640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0035019Z     outputs = self.model(
2025-09-07T08:06:04.0035379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0035757Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0036132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0036513Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0036853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0037203Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0037574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T08:06:04.0037959Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:06:04.0038343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0038722Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0038850Z 
2025-09-07T08:06:04.0038949Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0039365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0039679Z     return mod(**inputs)
2025-09-07T08:06:04.0040029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0040405Z     outputs = self.model(
2025-09-07T08:06:04.0040745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0041121Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0041483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0041859Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0042188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0042541Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0042973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:04.0043405Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0043573Z 
2025-09-07T08:06:04.0043674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0044020Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0044338Z     return mod(**inputs)
2025-09-07T08:06:04.0044698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0045075Z     outputs = self.model(
2025-09-07T08:06:04.0045422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0045794Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0046165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0046544Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0046881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0047226Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0047595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T08:06:04.0048014Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0048394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0048733Z     return self.act(input)
2025-09-07T08:06:04.0048839Z 
2025-09-07T08:06:04.0048933Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0049277Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0049598Z     return mod(**inputs)
2025-09-07T08:06:04.0049951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0050317Z     outputs = self.model(
2025-09-07T08:06:04.0050664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T08:06:04.0051038Z     encoder_outputs = self.encoder(
2025-09-07T08:06:04.0051408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T08:06:04.0051782Z     layer_outputs = encoder_layer(
2025-09-07T08:06:04.0052112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0052471Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0052846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T08:06:04.0053310Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0053441Z 
2025-09-07T08:06:04.0053540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0053875Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0054190Z     return mod(**inputs)
2025-09-07T08:06:04.0054535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0054907Z     outputs = self.model(
2025-09-07T08:06:04.0055249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0055626Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0055990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0056444Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0056780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0057130Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0057504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0057975Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0058377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0058834Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0059039Z 
2025-09-07T08:06:04.0059137Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0059482Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0059802Z     return mod(**inputs)
2025-09-07T08:06:04.0060155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0060527Z     outputs = self.model(
2025-09-07T08:06:04.0060877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0061255Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0061618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0061993Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0062323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0062673Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0063047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0063456Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0063844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0064232Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0064366Z 
2025-09-07T08:06:04.0064462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0064810Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0065126Z     return mod(**inputs)
2025-09-07T08:06:04.0065464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0065833Z     outputs = self.model(
2025-09-07T08:06:04.0066178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0066639Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0067005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0067371Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0067700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0068041Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0068399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0068782Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0069164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0069545Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0069679Z 
2025-09-07T08:06:04.0069757Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0070017Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0070207Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0070391Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0070598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0070931Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0071232Z     return mod(**inputs)
2025-09-07T08:06:04.0071569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0071933Z     outputs = self.model(
2025-09-07T08:06:04.0072270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0072638Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0072990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0073375Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0073709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0074050Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0074408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0074795Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0075184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0075580Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0076028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0076504Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0076693Z 
2025-09-07T08:06:04.0076786Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0077122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0077438Z     return mod(**inputs)
2025-09-07T08:06:04.0077781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0078150Z     outputs = self.model(
2025-09-07T08:06:04.0078487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0078858Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0079218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0079722Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0080140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0093527Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0094006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0094462Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0094897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0095325Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0095781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0096244Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0096408Z 
2025-09-07T08:06:04.0096510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0096999Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0097343Z     return mod(**inputs)
2025-09-07T08:06:04.0097767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0098158Z     outputs = self.model(
2025-09-07T08:06:04.0098508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0098888Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0099256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0099628Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0099970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0100341Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0100732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0101130Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0101523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0101899Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0102028Z 
2025-09-07T08:06:04.0102131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0102480Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0102794Z     return mod(**inputs)
2025-09-07T08:06:04.0103304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0103678Z     outputs = self.model(
2025-09-07T08:06:04.0104027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0104401Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0104764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0105133Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0105467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0105814Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0106180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0106587Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0106987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0107625Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0107830Z 
2025-09-07T08:06:04.0107932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0108280Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0108593Z     return mod(**inputs)
2025-09-07T08:06:04.0108942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0109317Z     outputs = self.model(
2025-09-07T08:06:04.0109664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0110049Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0110417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0110799Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0111235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0111599Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0111980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0112392Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0112800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0113186Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0113313Z 
2025-09-07T08:06:04.0113413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0113767Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0114082Z     return mod(**inputs)
2025-09-07T08:06:04.0114435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0114802Z     outputs = self.model(
2025-09-07T08:06:04.0115151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0115527Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0115893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0116281Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0116614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0116967Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0117341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0117743Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0118155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0118534Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0118677Z 
2025-09-07T08:06:04.0118755Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0118955Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0119150Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0119334Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0119553Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0119894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0120214Z     return mod(**inputs)
2025-09-07T08:06:04.0120558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0120927Z     outputs = self.model(
2025-09-07T08:06:04.0121345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0121724Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0122084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0122455Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0122786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0123139Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0123507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0123910Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0124300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0124794Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0125246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0125729Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0125910Z 
2025-09-07T08:06:04.0126009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0126348Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0126662Z     return mod(**inputs)
2025-09-07T08:06:04.0127006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0127381Z     outputs = self.model(
2025-09-07T08:06:04.0127719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0128103Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0128466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0128843Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0129173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0129520Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0129889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0130291Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0130691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0131074Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0131509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0131962Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0132117Z 
2025-09-07T08:06:04.0132211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0132548Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0132851Z     return mod(**inputs)
2025-09-07T08:06:04.0133192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0133254Z     outputs = self.model(
2025-09-07T08:06:04.0133502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0133567Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0133819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0133956Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0134171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0134244Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0134488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0134585Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0134826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0134898Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0134903Z 
2025-09-07T08:06:04.0134996Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0135184Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0135305Z     return mod(**inputs)
2025-09-07T08:06:04.0135557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0135623Z     outputs = self.model(
2025-09-07T08:06:04.0135871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0135940Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0136187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0136252Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0136471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0136541Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0136787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0136910Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0136914Z 
2025-09-07T08:06:04.0137006Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0137196Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0137254Z     return mod(**inputs)
2025-09-07T08:06:04.0137499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0137609Z     outputs = self.model(
2025-09-07T08:06:04.0137855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0137923Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0138168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0138247Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0138465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0138534Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0138788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0138901Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0139119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0139184Z     return self.act(input)
2025-09-07T08:06:04.0139187Z 
2025-09-07T08:06:04.0139280Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0139474Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0139533Z     return mod(**inputs)
2025-09-07T08:06:04.0139859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0139921Z     outputs = self.model(
2025-09-07T08:06:04.0140168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0140244Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0140495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0140567Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0140782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0140860Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0141103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T08:06:04.0141257Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0141261Z 
2025-09-07T08:06:04.0141362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0141550Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0141616Z     return mod(**inputs)
2025-09-07T08:06:04.0141860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0141923Z     outputs = self.model(
2025-09-07T08:06:04.0142175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0142240Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0142491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0142557Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0142779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0142856Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0143097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0143199Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0143443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0143594Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0143597Z 
2025-09-07T08:06:04.0143691Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0143879Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0143946Z     return mod(**inputs)
2025-09-07T08:06:04.0144196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0144268Z     outputs = self.model(
2025-09-07T08:06:04.0144515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0144580Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0144831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0144895Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0145119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0145188Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0145439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0145592Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0145837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0145917Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0145921Z 
2025-09-07T08:06:04.0146014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0146208Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0146266Z     return mod(**inputs)
2025-09-07T08:06:04.0146514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0146582Z     outputs = self.model(
2025-09-07T08:06:04.0146827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0146901Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0147207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0147274Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0147498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0147569Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0147825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0147909Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0148161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0148240Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0148244Z 
2025-09-07T08:06:04.0148321Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0148402Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0148476Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0148555Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0148646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0148832Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0148898Z     return mod(**inputs)
2025-09-07T08:06:04.0149143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0149213Z     outputs = self.model(
2025-09-07T08:06:04.0149457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0149522Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0149771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0149839Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0150063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0150132Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0150380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0150472Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0150715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0150811Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0151109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0151243Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0151303Z 
2025-09-07T08:06:04.0151403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0151597Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0151664Z     return mod(**inputs)
2025-09-07T08:06:04.0151919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0151987Z     outputs = self.model(
2025-09-07T08:06:04.0152235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0152303Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0152555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0152622Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0152849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0152981Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0153239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0153331Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0153575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0153670Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0153965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0154073Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0154077Z 
2025-09-07T08:06:04.0154171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0154361Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0154436Z     return mod(**inputs)
2025-09-07T08:06:04.0154682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0154754Z     outputs = self.model(
2025-09-07T08:06:04.0154999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0155071Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0155318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0155383Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0155608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0155683Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0155930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0156019Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0156263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0156334Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0156337Z 
2025-09-07T08:06:04.0156436Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0156620Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0156680Z     return mod(**inputs)
2025-09-07T08:06:04.0156931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0156992Z     outputs = self.model(
2025-09-07T08:06:04.0157244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0157372Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0157625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0157691Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0157905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0157975Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0158217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0158317Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0158569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0158718Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0158725Z 
2025-09-07T08:06:04.0158890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0159093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0159165Z     return mod(**inputs)
2025-09-07T08:06:04.0159420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0159485Z     outputs = self.model(
2025-09-07T08:06:04.0159731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0159799Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0160048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0160115Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0160344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0160421Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0160670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0160780Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0161022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0161098Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0161102Z 
2025-09-07T08:06:04.0161197Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0161388Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0161452Z     return mod(**inputs)
2025-09-07T08:06:04.0161701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0161771Z     outputs = self.model(
2025-09-07T08:06:04.0162012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0162082Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0162324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0162389Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0162617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0162696Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0162944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0163042Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0163286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0163431Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0163434Z 
2025-09-07T08:06:04.0163510Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0163583Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0163652Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0163721Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0163815Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0164003Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0164069Z     return mod(**inputs)
2025-09-07T08:06:04.0164313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0164375Z     outputs = self.model(
2025-09-07T08:06:04.0164701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0164772Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0165019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0165083Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0165302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0165370Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0165615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0165713Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0165956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0166048Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0166350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0166477Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0166480Z 
2025-09-07T08:06:04.0166574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0166760Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0166821Z     return mod(**inputs)
2025-09-07T08:06:04.0167068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0167134Z     outputs = self.model(
2025-09-07T08:06:04.0167377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0167442Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0167692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0167758Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0167975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0168042Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0168282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0168379Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0168621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0168707Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0168999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0169444Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0169452Z 
2025-09-07T08:06:04.0169545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0169731Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0169797Z     return mod(**inputs)
2025-09-07T08:06:04.0170044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0170108Z     outputs = self.model(
2025-09-07T08:06:04.0170350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0170415Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0170663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0170792Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0171014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0171084Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0171328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0171431Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0171670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0171749Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0171753Z 
2025-09-07T08:06:04.0171844Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0172035Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0172096Z     return mod(**inputs)
2025-09-07T08:06:04.0172344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0172409Z     outputs = self.model(
2025-09-07T08:06:04.0172650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0172720Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0172963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0173030Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0173250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0173319Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0173564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0173683Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0173687Z 
2025-09-07T08:06:04.0173780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0173966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0174025Z     return mod(**inputs)
2025-09-07T08:06:04.0174276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0174338Z     outputs = self.model(
2025-09-07T08:06:04.0174589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0174654Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0174898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0174966Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0175257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0175334Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0175584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0175696Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0175911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0175977Z     return self.act(input)
2025-09-07T08:06:04.0175980Z 
2025-09-07T08:06:04.0176076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0176263Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0176323Z     return mod(**inputs)
2025-09-07T08:06:04.0176632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0176695Z     outputs = self.model(
2025-09-07T08:06:04.0176942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0177007Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0177253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0177319Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0177598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0177674Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0177917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T08:06:04.0177994Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0178001Z 
2025-09-07T08:06:04.0178099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0178284Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0178347Z     return mod(**inputs)
2025-09-07T08:06:04.0178590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0178654Z     outputs = self.model(
2025-09-07T08:06:04.0178899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0178967Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0179212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0179277Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0179494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0179564Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0179812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0179903Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0180145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0180288Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0180291Z 
2025-09-07T08:06:04.0180382Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0180567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0180624Z     return mod(**inputs)
2025-09-07T08:06:04.0180871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0180996Z     outputs = self.model(
2025-09-07T08:06:04.0181241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0181311Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0181554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0181621Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0181832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0181902Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0182144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0182232Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0182541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0182612Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0182616Z 
2025-09-07T08:06:04.0182705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0182893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0182952Z     return mod(**inputs)
2025-09-07T08:06:04.0183198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0183256Z     outputs = self.model(
2025-09-07T08:06:04.0183503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0183569Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0183814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0183883Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0184093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0184161Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0184402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0184485Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0184730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0184805Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0184809Z 
2025-09-07T08:06:04.0184883Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0184951Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0185020Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0185096Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0185187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0185375Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0185433Z     return mod(**inputs)
2025-09-07T08:06:04.0185676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0185739Z     outputs = self.model(
2025-09-07T08:06:04.0185983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0186053Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0186297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0186362Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0186656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0186729Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0186977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0187064Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0187307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0187393Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0187689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0187818Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0187823Z 
2025-09-07T08:06:04.0187918Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0188190Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0188249Z     return mod(**inputs)
2025-09-07T08:06:04.0188496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0188561Z     outputs = self.model(
2025-09-07T08:06:04.0188803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0188873Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0189115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0189181Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0189393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0189460Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0189709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0189793Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0190054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0190139Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0190427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0190529Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0190533Z 
2025-09-07T08:06:04.0190622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0190808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0190869Z     return mod(**inputs)
2025-09-07T08:06:04.0191117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0191175Z     outputs = self.model(
2025-09-07T08:06:04.0191417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0191482Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0191724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0191789Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0192001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0192068Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0192311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0192460Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0192706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0192779Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0192782Z 
2025-09-07T08:06:04.0192878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0193065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0193124Z     return mod(**inputs)
2025-09-07T08:06:04.0193376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0193436Z     outputs = self.model(
2025-09-07T08:06:04.0193684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0193749Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0194054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0194126Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0194341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0194414Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0194658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0194758Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0195006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0195144Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0195148Z 
2025-09-07T08:06:04.0195251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0195436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0195501Z     return mod(**inputs)
2025-09-07T08:06:04.0195747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0195805Z     outputs = self.model(
2025-09-07T08:06:04.0196054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0196119Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0196371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0196435Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0196646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0196721Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0196961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0197060Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0197301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0197372Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0197377Z 
2025-09-07T08:06:04.0197466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0197648Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0197709Z     return mod(**inputs)
2025-09-07T08:06:04.0197953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0198075Z     outputs = self.model(
2025-09-07T08:06:04.0198322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0198387Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0198631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0198693Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0198907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0198974Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0199217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0199317Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0199574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0199716Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0199720Z 
2025-09-07T08:06:04.0199795Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0199871Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0199939Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0200008Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0200106Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0200298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0200362Z     return mod(**inputs)
2025-09-07T08:06:04.0200609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0200670Z     outputs = self.model(
2025-09-07T08:06:04.0200919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0200989Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0201236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0201301Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0201516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0201590Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0201832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0201930Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0202171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0202257Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0202556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0202679Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0202682Z 
2025-09-07T08:06:04.0202779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0202961Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0203021Z     return mod(**inputs)
2025-09-07T08:06:04.0203393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0203458Z     outputs = self.model(
2025-09-07T08:06:04.0203711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0203777Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0204030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0204191Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0204408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0204482Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0204729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0204833Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0205076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0205166Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0205457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0205644Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0205647Z 
2025-09-07T08:06:04.0205750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0205936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0206002Z     return mod(**inputs)
2025-09-07T08:06:04.0206250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0206312Z     outputs = self.model(
2025-09-07T08:06:04.0206564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0206632Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0206883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0206950Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0207169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0207242Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0207483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0207580Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0207821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0207898Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0207901Z 
2025-09-07T08:06:04.0207993Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0208178Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0208242Z     return mod(**inputs)
2025-09-07T08:06:04.0208493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0208559Z     outputs = self.model(
2025-09-07T08:06:04.0208804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0208870Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0209115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0209179Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0209397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0209468Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0209709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0209821Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0209904Z 
2025-09-07T08:06:04.0209995Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0210182Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0210240Z     return mod(**inputs)
2025-09-07T08:06:04.0210490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0210550Z     outputs = self.model(
2025-09-07T08:06:04.0210795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0210868Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0211116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0211191Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0211461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0211533Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0211787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0211891Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0212107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0212171Z     return self.act(input)
2025-09-07T08:06:04.0212174Z 
2025-09-07T08:06:04.0212271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0212457Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0212516Z     return mod(**inputs)
2025-09-07T08:06:04.0212767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0212833Z     outputs = self.model(
2025-09-07T08:06:04.0213082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0213148Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0213394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0213468Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0213681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0213755Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0213997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T08:06:04.0214072Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0214080Z 
2025-09-07T08:06:04.0214178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0214364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0214431Z     return mod(**inputs)
2025-09-07T08:06:04.0214675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0214742Z     outputs = self.model(
2025-09-07T08:06:04.0214986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0215052Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0215300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0215367Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0215585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0215718Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0215964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0216058Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0216301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0216449Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0216453Z 
2025-09-07T08:06:04.0216544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0216732Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0216792Z     return mod(**inputs)
2025-09-07T08:06:04.0217038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0217108Z     outputs = self.model(
2025-09-07T08:06:04.0217410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0217480Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0217759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0217826Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0218047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0218117Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0218366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0218456Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0218703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0218783Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0218786Z 
2025-09-07T08:06:04.0218880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0219072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0219131Z     return mod(**inputs)
2025-09-07T08:06:04.0219382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0219442Z     outputs = self.model(
2025-09-07T08:06:04.0219686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0219759Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0220009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0220089Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0220305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0220377Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0220623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0220709Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0220956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0221033Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0221036Z 
2025-09-07T08:06:04.0221113Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0221186Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0221255Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0221328Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0221532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0221720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0221787Z     return mod(**inputs)
2025-09-07T08:06:04.0222038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0222102Z     outputs = self.model(
2025-09-07T08:06:04.0222346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0222419Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0222663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0222728Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0222944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0223070Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0223332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0223423Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0223676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0223770Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0224069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0224198Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0224201Z 
2025-09-07T08:06:04.0224297Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0224495Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0224559Z     return mod(**inputs)
2025-09-07T08:06:04.0224810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0224875Z     outputs = self.model(
2025-09-07T08:06:04.0225120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0225190Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0225433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0225500Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0225721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0225791Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0226046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0226135Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0226379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0226469Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0226758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0226864Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0226868Z 
2025-09-07T08:06:04.0226962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0227157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0227217Z     return mod(**inputs)
2025-09-07T08:06:04.0227470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0227602Z     outputs = self.model(
2025-09-07T08:06:04.0227858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0227932Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0228181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0228248Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0228473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0228547Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0228801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0228894Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0229218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0229297Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0229300Z 
2025-09-07T08:06:04.0229396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0229606Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0229668Z     return mod(**inputs)
2025-09-07T08:06:04.0229922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0229985Z     outputs = self.model(
2025-09-07T08:06:04.0230233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0230307Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0230553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0230629Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0230846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0230917Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0231167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0231269Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0231516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0231659Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0231663Z 
2025-09-07T08:06:04.0231764Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0231963Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0232023Z     return mod(**inputs)
2025-09-07T08:06:04.0232275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0232335Z     outputs = self.model(
2025-09-07T08:06:04.0232587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0232652Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0232897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0232968Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0233183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0233260Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0233573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0233680Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0233923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0233998Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0234002Z 
2025-09-07T08:06:04.0234100Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0234292Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0234356Z     return mod(**inputs)
2025-09-07T08:06:04.0234602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0234663Z     outputs = self.model(
2025-09-07T08:06:04.0234975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0235044Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0235297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0235362Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0235578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0235655Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0235900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0236000Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0236244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0236334Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0236338Z 
2025-09-07T08:06:04.0236414Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0236488Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0236564Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0236633Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0236731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0236915Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0236975Z     return mod(**inputs)
2025-09-07T08:06:04.0237226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0237287Z     outputs = self.model(
2025-09-07T08:06:04.0237651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0237718Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0237972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0238045Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0238259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0238334Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0238577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0238672Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0238924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0239017Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0239324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0239513Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0239517Z 
2025-09-07T08:06:04.0239615Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0239801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0239861Z     return mod(**inputs)
2025-09-07T08:06:04.0240114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0240174Z     outputs = self.model(
2025-09-07T08:06:04.0240428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0240494Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0240740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0240871Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0241094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0241172Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0241418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0241522Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0241766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0241854Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0242151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0242247Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0242254Z 
2025-09-07T08:06:04.0242356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0242546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0242611Z     return mod(**inputs)
2025-09-07T08:06:04.0242866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0242929Z     outputs = self.model(
2025-09-07T08:06:04.0243182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0243248Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0243501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0243569Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0243788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0243867Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0244109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0244209Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0244450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0244523Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0244526Z 
2025-09-07T08:06:04.0244621Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0244807Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0244874Z     return mod(**inputs)
2025-09-07T08:06:04.0245120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0245449Z     outputs = self.model(
2025-09-07T08:06:04.0245701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0245767Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0246019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0246085Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0246304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0246373Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0246619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0246739Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0246745Z 
2025-09-07T08:06:04.0246895Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0247091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0247150Z     return mod(**inputs)
2025-09-07T08:06:04.0247400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0247467Z     outputs = self.model(
2025-09-07T08:06:04.0247714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0247788Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0248033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0248099Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0248338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0248413Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0248670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0248775Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0248992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0249056Z     return self.act(input)
2025-09-07T08:06:04.0249060Z 
2025-09-07T08:06:04.0249149Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0249339Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0249397Z     return mod(**inputs)
2025-09-07T08:06:04.0249650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0249709Z     outputs = self.model(
2025-09-07T08:06:04.0249959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0250029Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0250275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0250349Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0250565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0250640Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0250884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T08:06:04.0250958Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0250962Z 
2025-09-07T08:06:04.0251059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0251311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0251380Z     return mod(**inputs)
2025-09-07T08:06:04.0251626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0251687Z     outputs = self.model(
2025-09-07T08:06:04.0251940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0252005Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0252252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0252315Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0252531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0252604Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0252922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0253020Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0253262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0253410Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0253414Z 
2025-09-07T08:06:04.0253509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0253694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0253755Z     return mod(**inputs)
2025-09-07T08:06:04.0253999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0254060Z     outputs = self.model(
2025-09-07T08:06:04.0254308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0254372Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0254618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0254683Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0254897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0254965Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0255208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0255297Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0255537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0255619Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0255622Z 
2025-09-07T08:06:04.0255714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0255905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0255965Z     return mod(**inputs)
2025-09-07T08:06:04.0256209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0256275Z     outputs = self.model(
2025-09-07T08:06:04.0256524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0256592Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0256836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0256903Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0257195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0257267Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0257594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0257683Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0257934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0258013Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0258016Z 
2025-09-07T08:06:04.0258090Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0258167Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0258237Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0258313Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0258402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0258654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0258721Z     return mod(**inputs)
2025-09-07T08:06:04.0258968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0259035Z     outputs = self.model(
2025-09-07T08:06:04.0259278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0259342Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0259592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0259657Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0259877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0259950Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0260194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0260286Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0260531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0260627Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0260924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0261053Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0261056Z 
2025-09-07T08:06:04.0261147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0261334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0261404Z     return mod(**inputs)
2025-09-07T08:06:04.0261653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0261720Z     outputs = self.model(
2025-09-07T08:06:04.0261965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0262029Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0262279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0262344Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0262562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0262629Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0262879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0263026Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0263271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0263362Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0263652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0263759Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0263763Z 
2025-09-07T08:06:04.0263855Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0264042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0264104Z     return mod(**inputs)
2025-09-07T08:06:04.0264349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0264475Z     outputs = self.model(
2025-09-07T08:06:04.0264727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0264792Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0265045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0265111Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0265330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0265399Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0265648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0265740Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0265989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0266073Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0266077Z 
2025-09-07T08:06:04.0266170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0266366Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0266428Z     return mod(**inputs)
2025-09-07T08:06:04.0266677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0266744Z     outputs = self.model(
2025-09-07T08:06:04.0266993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0267075Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0267320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0267400Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0267617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0267688Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0267941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0268042Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0268292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0268436Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0268440Z 
2025-09-07T08:06:04.0268537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0268739Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0268862Z     return mod(**inputs)
2025-09-07T08:06:04.0269117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0269179Z     outputs = self.model(
2025-09-07T08:06:04.0269426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0269506Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0269753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0269830Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0270046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0270123Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0270425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0270530Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0270779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0270854Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0270858Z 
2025-09-07T08:06:04.0270959Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0271147Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0271209Z     return mod(**inputs)
2025-09-07T08:06:04.0271466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0271528Z     outputs = self.model(
2025-09-07T08:06:04.0271779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0271851Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0272104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0272171Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0272386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0272463Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0272707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0272809Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0273054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0273132Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0273138Z 
2025-09-07T08:06:04.0273222Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0273295Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0273372Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0273442Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0273537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0273730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0273789Z     return mod(**inputs)
2025-09-07T08:06:04.0274042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0274102Z     outputs = self.model(
2025-09-07T08:06:04.0274349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0274424Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0274668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0274814Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0275029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0275106Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0275351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0275449Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0275698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0275786Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0276087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0276272Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0276276Z 
2025-09-07T08:06:04.0276373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0276568Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0276629Z     return mod(**inputs)
2025-09-07T08:06:04.0276880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0276941Z     outputs = self.model(
2025-09-07T08:06:04.0277193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0277259Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0277503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0277577Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0277796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0277875Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0278116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0278212Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0278463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0278547Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0278844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0278942Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0278945Z 
2025-09-07T08:06:04.0279050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0279239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0279300Z     return mod(**inputs)
2025-09-07T08:06:04.0279551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0279610Z     outputs = self.model(
2025-09-07T08:06:04.0279863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0279928Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0280172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0280243Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0280456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0280592Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0280834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0280928Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0281178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0281254Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0281257Z 
2025-09-07T08:06:04.0281357Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0281541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0281603Z     return mod(**inputs)
2025-09-07T08:06:04.0281847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0281912Z     outputs = self.model(
2025-09-07T08:06:04.0282218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0282287Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0282541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0282607Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0282822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0282898Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0283141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0283264Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0283268Z 
2025-09-07T08:06:04.0283360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0283552Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0283622Z     return mod(**inputs)
2025-09-07T08:06:04.0283864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0283930Z     outputs = self.model(
2025-09-07T08:06:04.0284174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0284247Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0284491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0284557Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0284774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0284843Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0285099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0285204Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0285415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0285485Z     return self.act(input)
2025-09-07T08:06:04.0285488Z 
2025-09-07T08:06:04.0285581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0285774Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0285832Z     return mod(**inputs)
2025-09-07T08:06:04.0286080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0286145Z     outputs = self.model(
2025-09-07T08:06:04.0286391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0286523Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0286768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0286840Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0287056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0287125Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0287373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T08:06:04.0287449Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0287452Z 
2025-09-07T08:06:04.0287551Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0287738Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0287854Z     return mod(**inputs)
2025-09-07T08:06:04.0288111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0288177Z     outputs = self.model(
2025-09-07T08:06:04.0288433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0288502Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0288750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0288827Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0289050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0289130Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0289383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0289486Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0289733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0289874Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0289877Z 
2025-09-07T08:06:04.0289978Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0290167Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0290235Z     return mod(**inputs)
2025-09-07T08:06:04.0290480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0290545Z     outputs = self.model(
2025-09-07T08:06:04.0290797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0290864Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0291118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0291183Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0291406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0291474Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0291718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0291816Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0292060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0292141Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0292202Z 
2025-09-07T08:06:04.0292298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0292489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0292556Z     return mod(**inputs)
2025-09-07T08:06:04.0292804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0292871Z     outputs = self.model(
2025-09-07T08:06:04.0293116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0293185Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0293436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0293502Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0293780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0293853Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0294103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0294191Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0294435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0294519Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0294522Z 
2025-09-07T08:06:04.0294596Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0294671Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0294742Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0294814Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0294912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0295100Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0295168Z     return mod(**inputs)
2025-09-07T08:06:04.0295415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0295475Z     outputs = self.model(
2025-09-07T08:06:04.0295727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0295795Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0296047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0296113Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0296329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0296407Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0296657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0296750Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0296992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0297085Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0297379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0297503Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0297506Z 
2025-09-07T08:06:04.0297637Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0297823Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0297890Z     return mod(**inputs)
2025-09-07T08:06:04.0298222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0298287Z     outputs = self.model(
2025-09-07T08:06:04.0298541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0298610Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0298862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0298927Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0299148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0299220Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0299466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0299621Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0299868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0299959Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0300248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0300347Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0300356Z 
2025-09-07T08:06:04.0300450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0300635Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0300702Z     return mod(**inputs)
2025-09-07T08:06:04.0300950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0301018Z     outputs = self.model(
2025-09-07T08:06:04.0301265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0301330Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0301580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0301646Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0301865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0301934Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0302176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T08:06:04.0302270Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T08:06:04.0302517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0302598Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0302601Z 
2025-09-07T08:06:04.0302694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0302887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0302948Z     return mod(**inputs)
2025-09-07T08:06:04.0303332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0303404Z     outputs = self.model(
2025-09-07T08:06:04.0303663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0303736Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0303983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0304147Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0304378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0304451Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0304709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0304811Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0305056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T08:06:04.0305208Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T08:06:04.0305212Z 
2025-09-07T08:06:04.0305307Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0305507Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0305571Z     return mod(**inputs)
2025-09-07T08:06:04.0305908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0305971Z     outputs = self.model(
2025-09-07T08:06:04.0306217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0306289Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0306536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0306608Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0306824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0306895Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0307147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0307249Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0307499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T08:06:04.0307573Z     key_states = self.k_proj(current_states)
2025-09-07T08:06:04.0307577Z 
2025-09-07T08:06:04.0307673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0307860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0307918Z     return mod(**inputs)
2025-09-07T08:06:04.0308172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0308233Z     outputs = self.model(
2025-09-07T08:06:04.0308487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0308559Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0308805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0308877Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0309090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0309165Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0309408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0309504Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0309755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T08:06:04.0309833Z     value_states = self.v_proj(current_states)
2025-09-07T08:06:04.0309836Z 
2025-09-07T08:06:04.0309916Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0310046Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0310117Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0310189Z cudagraph partition due to non gpu ops
2025-09-07T08:06:04.0310278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0310469Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0310528Z     return mod(**inputs)
2025-09-07T08:06:04.0310774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0310835Z     outputs = self.model(
2025-09-07T08:06:04.0311080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0311149Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0311390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0311519Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0311741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0311815Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0312064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0312164Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0312411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0312498Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0312792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:06:04.0312917Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:04.0312925Z 
2025-09-07T08:06:04.0313020Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0313213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0313271Z     return mod(**inputs)
2025-09-07T08:06:04.0313522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0313580Z     outputs = self.model(
2025-09-07T08:06:04.0313825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0313892Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0314134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0314201Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0314420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0314492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0314738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0314836Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0315082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T08:06:04.0315164Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:06:04.0315453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:06:04.0315550Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:06:04.0315554Z 
2025-09-07T08:06:04.0315644Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0315933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0315992Z     return mod(**inputs)
2025-09-07T08:06:04.0316241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0316302Z     outputs = self.model(
2025-09-07T08:06:04.0316545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0316610Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0316855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0316926Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0317140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0317211Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0317542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T08:06:04.0317640Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T08:06:04.0317889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T08:06:04.0317964Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:06:04.0317967Z 
2025-09-07T08:06:04.0318062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0318249Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0318311Z     return mod(**inputs)
2025-09-07T08:06:04.0318563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0318623Z     outputs = self.model(
2025-09-07T08:06:04.0318914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0318980Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0319225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0319297Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0319510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0319585Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0319826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0319942Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0319945Z 
2025-09-07T08:06:04.0320038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0320228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0320296Z     return mod(**inputs)
2025-09-07T08:06:04.0320543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0320609Z     outputs = self.model(
2025-09-07T08:06:04.0320851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0320917Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0321168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0321234Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0321453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0321521Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0321766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T08:06:04.0321936Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:06:04.0322148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:04.0322220Z     return self.act(input)
2025-09-07T08:06:04.0322224Z 
2025-09-07T08:06:04.0322313Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0322504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0322565Z     return mod(**inputs)
2025-09-07T08:06:04.0322812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T08:06:04.0327928Z     outputs = self.model(
2025-09-07T08:06:04.0328255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T08:06:04.0328433Z     decoder_outputs = self.decoder(
2025-09-07T08:06:04.0328725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T08:06:04.0328798Z     layer_outputs = decoder_layer(
2025-09-07T08:06:04.0329045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:04.0329124Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:04.0329387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T08:06:04.0329465Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:06:04.0329469Z 
2025-09-07T08:06:04.0329571Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0329775Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0329843Z     return mod(**inputs)
2025-09-07T08:06:04.0330107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1490, in forward
2025-09-07T08:06:04.0330182Z     lm_logits = self.lm_head(outputs[0])
2025-09-07T08:06:04.0330186Z 
2025-09-07T08:06:04.0330281Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:04.0330471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:04.0330529Z     return mod(**inputs)
2025-09-07T08:06:04.0330782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1491, in forward
2025-09-07T08:06:04.0330905Z     lm_logits = lm_logits + self.final_logits_bias.to(lm_logits.device)
2025-09-07T08:06:04.0330909Z 
2025-09-07T08:06:16.7290774Z pass
2025-09-07T08:06:16.7291180Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:19.4592140Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:06:19.4593046Z   import pynvml  # type: ignore[import]
2025-09-07T08:06:21.4476638Z 
2025-09-07T08:06:24.5647242Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:06:24.5647892Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:06:24.5887106Z cpu  eval  hf_Bert                            
2025-09-07T08:06:25.6182284Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:25.9062239Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:26.1801367Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:33.2813419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2814485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2814849Z     return mod(**inputs)
2025-09-07T08:06:33.2815269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2815674Z     outputs = self.bert(
2025-09-07T08:06:33.2816045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 964, in forward
2025-09-07T08:06:33.2816472Z     embedding_output = self.embeddings(
2025-09-07T08:06:33.2816951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 188, in forward
2025-09-07T08:06:33.2817348Z     embeddings = self.LayerNorm(embeddings)
2025-09-07T08:06:33.2817495Z 
2025-09-07T08:06:33.2817614Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2817826Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2818169Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2818364Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2818564Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2818758Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2818954Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2819143Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2819338Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2819550Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2819766Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2819951Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2820187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2820563Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2820894Z     return mod(**inputs)
2025-09-07T08:06:33.2821274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2821667Z     outputs = self.bert(
2025-09-07T08:06:33.2822024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2822445Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2822839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2823230Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2823597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2823971Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2824364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2824762Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2825167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2825561Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2825930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2826365Z     self_outputs = self.self(
2025-09-07T08:06:33.2826740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2827112Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2827467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.2828012Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.2828293Z 
2025-09-07T08:06:33.2828399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2828842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2829169Z     return mod(**inputs)
2025-09-07T08:06:33.2829541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2829923Z     outputs = self.bert(
2025-09-07T08:06:33.2830278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2830681Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2831054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2831435Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2831798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2832167Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2832640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2833065Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2833434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2833805Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2834168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2834547Z     self_outputs = self.self(
2025-09-07T08:06:33.2834896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2835265Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2835632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.2836022Z     self.key(current_states)
2025-09-07T08:06:33.2836133Z 
2025-09-07T08:06:33.2836234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2836581Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2836905Z     return mod(**inputs)
2025-09-07T08:06:33.2837259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2837629Z     outputs = self.bert(
2025-09-07T08:06:33.2837970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2838360Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2838728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2839103Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2839446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2839795Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2840174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2840558Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2840940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2841308Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2841658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2842030Z     self_outputs = self.self(
2025-09-07T08:06:33.2842384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2842835Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2843195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.2843571Z     self.value(current_states)
2025-09-07T08:06:33.2843689Z 
2025-09-07T08:06:33.2843768Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2844005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2844351Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2844658Z     return mod(**inputs)
2025-09-07T08:06:33.2845009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2845383Z     outputs = self.bert(
2025-09-07T08:06:33.2845726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2846102Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2846535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2846913Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2847260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2847627Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2847999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2848388Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2848760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2849125Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2849478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2849871Z     self_outputs = self.self(
2025-09-07T08:06:33.2850215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2850589Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2850941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.2851376Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.2851560Z 
2025-09-07T08:06:33.2851664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2852012Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2852332Z     return mod(**inputs)
2025-09-07T08:06:33.2852687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2853069Z     outputs = self.bert(
2025-09-07T08:06:33.2853424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2853799Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2854177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2854557Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2854898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2855251Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2855633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2856014Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2856388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2856840Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2857203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.2857704Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.2858132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.2858521Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2858657Z 
2025-09-07T08:06:33.2858762Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2859114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2859450Z     return mod(**inputs)
2025-09-07T08:06:33.2859815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2860204Z     outputs = self.bert(
2025-09-07T08:06:33.2860628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2861020Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2861400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2861786Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2862130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2862481Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2862868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2863258Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2863665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2864065Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2864477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.2864934Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.2865358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.2865746Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2865878Z 
2025-09-07T08:06:33.2865979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2866336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2866658Z     return mod(**inputs)
2025-09-07T08:06:33.2867017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2867398Z     outputs = self.bert(
2025-09-07T08:06:33.2867739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2868124Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2868494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2868870Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2869203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2869574Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2869948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2870333Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2870730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2871200Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2871606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.2872062Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.2872482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.2872897Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.2873272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.2873614Z     return self.act(input)
2025-09-07T08:06:33.2873728Z 
2025-09-07T08:06:33.2873825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2874244Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2874562Z     return mod(**inputs)
2025-09-07T08:06:33.2874914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2875287Z     outputs = self.bert(
2025-09-07T08:06:33.2875635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2876014Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2876377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2876755Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2877092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2877447Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2877828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2878212Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2878612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2879004Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2879409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.2879869Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.2880304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.2880695Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2880831Z 
2025-09-07T08:06:33.2880938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2881295Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2881606Z     return mod(**inputs)
2025-09-07T08:06:33.2881962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2882335Z     outputs = self.bert(
2025-09-07T08:06:33.2882680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2883061Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2883426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2883802Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2884142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2884498Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2884942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2885330Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2885709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2886086Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2886446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2886814Z     self_outputs = self.self(
2025-09-07T08:06:33.2887168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2887536Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2887899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.2888480Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.2888752Z 
2025-09-07T08:06:33.2888855Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2889210Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2889528Z     return mod(**inputs)
2025-09-07T08:06:33.2889880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2890250Z     outputs = self.bert(
2025-09-07T08:06:33.2890599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2890980Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2891348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2891730Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2892066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2892427Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2892801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2893187Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2893573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2893952Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2894313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2894694Z     self_outputs = self.self(
2025-09-07T08:06:33.2895046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2895414Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2895770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.2896143Z     self.key(current_states)
2025-09-07T08:06:33.2896252Z 
2025-09-07T08:06:33.2896354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2896706Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2897018Z     return mod(**inputs)
2025-09-07T08:06:33.2897368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2897826Z     outputs = self.bert(
2025-09-07T08:06:33.2898176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2898545Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2898989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2899361Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2899704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2900280Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2900653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2901040Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2901413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2901782Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2902136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2902566Z     self_outputs = self.self(
2025-09-07T08:06:33.2902916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2903452Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2903824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.2904198Z     self.value(current_states)
2025-09-07T08:06:33.2904317Z 
2025-09-07T08:06:33.2904393Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2904627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2904990Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2905309Z     return mod(**inputs)
2025-09-07T08:06:33.2905663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2906045Z     outputs = self.bert(
2025-09-07T08:06:33.2906398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2906777Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2907147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2907529Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2907871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2908228Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2908605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2908987Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2909362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2909739Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2910101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2910469Z     self_outputs = self.self(
2025-09-07T08:06:33.2910817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2911184Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2911541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.2911972Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.2912154Z 
2025-09-07T08:06:33.2912254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2912608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2913085Z     return mod(**inputs)
2025-09-07T08:06:33.2913452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2913827Z     outputs = self.bert(
2025-09-07T08:06:33.2914169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2914556Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2914923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2915306Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2915640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2915998Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2916374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2917773Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2918165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2918527Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2918890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.2919318Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.2919743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.2920132Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2920267Z 
2025-09-07T08:06:33.2920368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2920725Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2921052Z     return mod(**inputs)
2025-09-07T08:06:33.2921488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2921865Z     outputs = self.bert(
2025-09-07T08:06:33.2922210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2922603Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2922969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2923343Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2923675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2924031Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2924402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2924801Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2925195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2925597Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2926010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.2926473Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.2926919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.2927315Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2927461Z 
2025-09-07T08:06:33.2927564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2927922Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2928332Z     return mod(**inputs)
2025-09-07T08:06:33.2928697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2929078Z     outputs = self.bert(
2025-09-07T08:06:33.2929443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2929842Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2930253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2930631Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2930980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2931350Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2931807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2932217Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2932619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2933020Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2933436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.2933902Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.2934333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.2934745Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.2935129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.2935473Z     return self.act(input)
2025-09-07T08:06:33.2935582Z 
2025-09-07T08:06:33.2935691Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2936052Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2936369Z     return mod(**inputs)
2025-09-07T08:06:33.2936728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2937107Z     outputs = self.bert(
2025-09-07T08:06:33.2937453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2937903Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2938280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2938661Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2939009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2939374Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2939743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2940134Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2940544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2940945Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2941351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.2941820Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.2942258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.2942731Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2942864Z 
2025-09-07T08:06:33.2942973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2943320Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2943639Z     return mod(**inputs)
2025-09-07T08:06:33.2943993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2944371Z     outputs = self.bert(
2025-09-07T08:06:33.2944720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2945100Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2945466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2945844Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2946244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2946610Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2947003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2947396Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2947782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2948155Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2948512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2948888Z     self_outputs = self.self(
2025-09-07T08:06:33.2949247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2949629Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2949983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.2950505Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.2950777Z 
2025-09-07T08:06:33.2950876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2951229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2951551Z     return mod(**inputs)
2025-09-07T08:06:33.2951897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2952273Z     outputs = self.bert(
2025-09-07T08:06:33.2952619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2953012Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2953377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2953747Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2954084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2954442Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2954824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2955207Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2955576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2955956Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2956322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2956769Z     self_outputs = self.self(
2025-09-07T08:06:33.2957117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2957482Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2957838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.2958214Z     self.key(current_states)
2025-09-07T08:06:33.2958325Z 
2025-09-07T08:06:33.2958427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2958775Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2959091Z     return mod(**inputs)
2025-09-07T08:06:33.2959448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2959834Z     outputs = self.bert(
2025-09-07T08:06:33.2960237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2960624Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2960993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2961372Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2961708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2962085Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2962460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2962844Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2963216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2963585Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2963939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2964315Z     self_outputs = self.self(
2025-09-07T08:06:33.2964667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2965037Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2965385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.2965762Z     self.value(current_states)
2025-09-07T08:06:33.2965881Z 
2025-09-07T08:06:33.2965960Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.2966193Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2966542Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2966859Z     return mod(**inputs)
2025-09-07T08:06:33.2967210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2967590Z     outputs = self.bert(
2025-09-07T08:06:33.2967935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2968309Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2968675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2969051Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2969387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2969739Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2970113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2970602Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2970975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2971345Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2971693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.2972069Z     self_outputs = self.self(
2025-09-07T08:06:33.2972416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2972785Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2973138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.2973563Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.2973757Z 
2025-09-07T08:06:33.2973912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2974263Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2974580Z     return mod(**inputs)
2025-09-07T08:06:33.2974931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2975302Z     outputs = self.bert(
2025-09-07T08:06:33.2975647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2976026Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2976390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2976762Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2977103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2977466Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2977892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.2978276Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.2978641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.2979008Z     return func(*args, **kwargs)
2025-09-07T08:06:33.2979361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.2979795Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.2980218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.2980604Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2980751Z 
2025-09-07T08:06:33.2980851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2981201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2981516Z     return mod(**inputs)
2025-09-07T08:06:33.2981857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2982228Z     outputs = self.bert(
2025-09-07T08:06:33.2982575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2982955Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2983330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2983697Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2984037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2984476Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2984853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2985234Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2985638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2986038Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2986463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.2986925Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.2987345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.2987738Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.2987935Z 
2025-09-07T08:06:33.2988034Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2988392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2988710Z     return mod(**inputs)
2025-09-07T08:06:33.2989056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2989430Z     outputs = self.bert(
2025-09-07T08:06:33.2989776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2990164Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2990525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2990903Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2991242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2991605Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2991983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.2992364Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.2992765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.2993159Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.2993566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.2994025Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.2994439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.2994860Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.2995240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.2995583Z     return self.act(input)
2025-09-07T08:06:33.2995688Z 
2025-09-07T08:06:33.2995785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.2996137Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.2996464Z     return mod(**inputs)
2025-09-07T08:06:33.2996819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.2997200Z     outputs = self.bert(
2025-09-07T08:06:33.2997546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.2997934Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.2998374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.2998755Z     layer_outputs = layer_module(
2025-09-07T08:06:33.2999086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.2999445Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.2999821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3000209Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3000609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3001001Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3001408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3001984Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3002431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3002822Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3002952Z 
2025-09-07T08:06:33.3003176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3003546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3003868Z     return mod(**inputs)
2025-09-07T08:06:33.3004220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3004593Z     outputs = self.bert(
2025-09-07T08:06:33.3004935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3005319Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3005688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3006066Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3006395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3006749Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3007128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3007513Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3007890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3008258Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3008618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3009000Z     self_outputs = self.self(
2025-09-07T08:06:33.3009351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3009715Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3010072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3010591Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3010861Z 
2025-09-07T08:06:33.3010958Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3011307Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3011622Z     return mod(**inputs)
2025-09-07T08:06:33.3011976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3012467Z     outputs = self.bert(
2025-09-07T08:06:33.3012827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3013209Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3013580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3013962Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3014312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3014674Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3015050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3015440Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3015900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3016286Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3016647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3017018Z     self_outputs = self.self(
2025-09-07T08:06:33.3017370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3017791Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3018150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3018527Z     self.key(current_states)
2025-09-07T08:06:33.3018639Z 
2025-09-07T08:06:33.3018737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3019084Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3019411Z     return mod(**inputs)
2025-09-07T08:06:33.3019764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3020135Z     outputs = self.bert(
2025-09-07T08:06:33.3020486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3020866Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3021236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3021617Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3021951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3022311Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3022684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3023078Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3023443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3023813Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3024168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3024543Z     self_outputs = self.self(
2025-09-07T08:06:33.3024892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3025253Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3025609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3025986Z     self.value(current_states)
2025-09-07T08:06:33.3026098Z 
2025-09-07T08:06:33.3026272Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3026501Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3026849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3027165Z     return mod(**inputs)
2025-09-07T08:06:33.3027517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3027897Z     outputs = self.bert(
2025-09-07T08:06:33.3028235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3028615Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3028983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3029358Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3029747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3030116Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3030491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3030880Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3031253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3031618Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3031972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3032351Z     self_outputs = self.self(
2025-09-07T08:06:33.3032699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3033063Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3033418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3033845Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3034034Z 
2025-09-07T08:06:33.3034130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3034474Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3034784Z     return mod(**inputs)
2025-09-07T08:06:33.3035135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3035510Z     outputs = self.bert(
2025-09-07T08:06:33.3035856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3036235Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3036598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3036979Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3037313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3037669Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3038043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3038421Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3038797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3039165Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3039518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3039940Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3040448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3040839Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3040974Z 
2025-09-07T08:06:33.3041075Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3041417Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3041727Z     return mod(**inputs)
2025-09-07T08:06:33.3042075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3042451Z     outputs = self.bert(
2025-09-07T08:06:33.3042800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3043184Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3043597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3043994Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3044338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3044698Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3045067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3045457Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3045856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3046255Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3046661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3047117Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3047537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3047923Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3048053Z 
2025-09-07T08:06:33.3048159Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3048505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3048822Z     return mod(**inputs)
2025-09-07T08:06:33.3049173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3049548Z     outputs = self.bert(
2025-09-07T08:06:33.3049891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3050267Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3050635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3051019Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3064352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3064748Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3065144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3065548Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3065965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3066360Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3066776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3067338Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3067758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3068171Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3068548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3068884Z     return self.act(input)
2025-09-07T08:06:33.3068988Z 
2025-09-07T08:06:33.3069089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3069442Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3069755Z     return mod(**inputs)
2025-09-07T08:06:33.3070103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3070471Z     outputs = self.bert(
2025-09-07T08:06:33.3070941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3071327Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3071698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3072066Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3072402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3072754Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3073122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3073500Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3073892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3074282Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3074680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3075140Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3075568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3075947Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3076079Z 
2025-09-07T08:06:33.3076174Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3076527Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3076836Z     return mod(**inputs)
2025-09-07T08:06:33.3077179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3077551Z     outputs = self.bert(
2025-09-07T08:06:33.3077887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3078256Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3078613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3079097Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3079429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3079779Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3080148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3080525Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3080893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3081345Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3081698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3082063Z     self_outputs = self.self(
2025-09-07T08:06:33.3082411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3082786Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3083153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3083666Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3083930Z 
2025-09-07T08:06:33.3084032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3084443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3084760Z     return mod(**inputs)
2025-09-07T08:06:33.3085116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3085482Z     outputs = self.bert(
2025-09-07T08:06:33.3085818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3086191Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3086560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3086927Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3087258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3087605Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3087975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3088353Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3088717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3089078Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3089422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3089782Z     self_outputs = self.self(
2025-09-07T08:06:33.3090124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3090479Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3090822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3091183Z     self.key(current_states)
2025-09-07T08:06:33.3091292Z 
2025-09-07T08:06:33.3091390Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3091731Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3092038Z     return mod(**inputs)
2025-09-07T08:06:33.3092375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3092733Z     outputs = self.bert(
2025-09-07T08:06:33.3093062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3093430Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3093780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3094137Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3094465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3094892Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3095268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3095646Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3096032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3096400Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3096750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3097112Z     self_outputs = self.self(
2025-09-07T08:06:33.3097452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3097878Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3098296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3098670Z     self.value(current_states)
2025-09-07T08:06:33.3098778Z 
2025-09-07T08:06:33.3098854Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3099071Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3099411Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3099714Z     return mod(**inputs)
2025-09-07T08:06:33.3100052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3100416Z     outputs = self.bert(
2025-09-07T08:06:33.3100755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3101123Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3101485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3101855Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3102177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3102540Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3102904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3103428Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3103793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3104149Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3104496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3104862Z     self_outputs = self.self(
2025-09-07T08:06:33.3105219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3105573Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3105915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3106335Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3106513Z 
2025-09-07T08:06:33.3106608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3106946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3107249Z     return mod(**inputs)
2025-09-07T08:06:33.3107592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3107955Z     outputs = self.bert(
2025-09-07T08:06:33.3108289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3108785Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3109136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3109495Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3109817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3110157Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3110514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3110882Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3111246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3111597Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3112261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3112689Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3113099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3113475Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3113607Z 
2025-09-07T08:06:33.3113703Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3114044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3114347Z     return mod(**inputs)
2025-09-07T08:06:33.3114688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3115054Z     outputs = self.bert(
2025-09-07T08:06:33.3115396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3115767Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3116122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3116483Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3116810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3117154Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3117513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3117889Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3118283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3118673Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3119071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3119519Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3119930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3120299Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3120428Z 
2025-09-07T08:06:33.3120526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3120861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3121166Z     return mod(**inputs)
2025-09-07T08:06:33.3121505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3121868Z     outputs = self.bert(
2025-09-07T08:06:33.3122276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3122644Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3122998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3123362Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3123691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3124033Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3124395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3124767Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3125149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3125591Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3125982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3126419Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3126822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3127225Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3127592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3127918Z     return self.act(input)
2025-09-07T08:06:33.3128023Z 
2025-09-07T08:06:33.3128114Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3128450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3128756Z     return mod(**inputs)
2025-09-07T08:06:33.3129088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3129454Z     outputs = self.bert(
2025-09-07T08:06:33.3129800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3130170Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3130540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3130908Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3131239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3131597Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3131976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3132368Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3132763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3133155Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3133557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3134020Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3134449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3134833Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3134970Z 
2025-09-07T08:06:33.3135068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3135420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3135828Z     return mod(**inputs)
2025-09-07T08:06:33.3136177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3136550Z     outputs = self.bert(
2025-09-07T08:06:33.3136891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3137263Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3137705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3138073Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3138406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3138762Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3139204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3139589Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3139963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3140332Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3140688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3141057Z     self_outputs = self.self(
2025-09-07T08:06:33.3141404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3141776Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3142145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3142658Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3142923Z 
2025-09-07T08:06:33.3143029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3143372Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3143689Z     return mod(**inputs)
2025-09-07T08:06:33.3144042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3144408Z     outputs = self.bert(
2025-09-07T08:06:33.3144738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3145112Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3145478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3145853Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3146191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3146534Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3146902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3147272Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3147638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3147992Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3148343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3148705Z     self_outputs = self.self(
2025-09-07T08:06:33.3149050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3149480Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3149826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3150199Z     self.key(current_states)
2025-09-07T08:06:33.3150313Z 
2025-09-07T08:06:33.3150409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3150758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3151071Z     return mod(**inputs)
2025-09-07T08:06:33.3151412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3151779Z     outputs = self.bert(
2025-09-07T08:06:33.3152116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3152490Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3152906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3153293Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3153625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3153983Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3154353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3154730Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3155107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3155467Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3155819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3156187Z     self_outputs = self.self(
2025-09-07T08:06:33.3156545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3156906Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3157254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3157623Z     self.value(current_states)
2025-09-07T08:06:33.3157732Z 
2025-09-07T08:06:33.3157808Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3158029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3158376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3158688Z     return mod(**inputs)
2025-09-07T08:06:33.3159026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3159397Z     outputs = self.bert(
2025-09-07T08:06:33.3159741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3160118Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3160477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3160844Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3161170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3161519Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3161891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3162268Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3162626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3163064Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3163427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3163804Z     self_outputs = self.self(
2025-09-07T08:06:33.3164150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3164517Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3164869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3165297Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3165474Z 
2025-09-07T08:06:33.3165581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3165923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3166232Z     return mod(**inputs)
2025-09-07T08:06:33.3166640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3167016Z     outputs = self.bert(
2025-09-07T08:06:33.3167354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3167734Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3168097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3168466Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3168798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3169143Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3169510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3169887Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3170260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3170625Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3170979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3171406Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3171837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3172231Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3172361Z 
2025-09-07T08:06:33.3172454Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3172798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3173113Z     return mod(**inputs)
2025-09-07T08:06:33.3173462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3173833Z     outputs = self.bert(
2025-09-07T08:06:33.3174169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3174547Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3174908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3175279Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3175601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3175947Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3176314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3176776Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3177169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3177619Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3178028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3178483Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3178897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3179276Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3179407Z 
2025-09-07T08:06:33.3179503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3179844Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3180153Z     return mod(**inputs)
2025-09-07T08:06:33.3180569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3180946Z     outputs = self.bert(
2025-09-07T08:06:33.3181285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3181657Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3182018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3182390Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3182715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3183062Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3183429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3183817Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3184208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3184588Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3184988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3185433Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3185845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3186256Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3186622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3186958Z     return self.act(input)
2025-09-07T08:06:33.3187067Z 
2025-09-07T08:06:33.3187169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3187519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3187825Z     return mod(**inputs)
2025-09-07T08:06:33.3188170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3188544Z     outputs = self.bert(
2025-09-07T08:06:33.3188887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3189262Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3189618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3189989Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3190318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3190746Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3191116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3191498Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3191892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3192280Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3192676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3193131Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3193559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3193940Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3194132Z 
2025-09-07T08:06:33.3194232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3194578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3194884Z     return mod(**inputs)
2025-09-07T08:06:33.3195230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3195601Z     outputs = self.bert(
2025-09-07T08:06:33.3195941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3196309Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3196670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3197037Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3197379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3197736Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3198126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3198508Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3198885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3199252Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3199619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3200006Z     self_outputs = self.self(
2025-09-07T08:06:33.3200354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3200715Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3201071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3201572Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3201841Z 
2025-09-07T08:06:33.3201936Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3202274Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3202585Z     return mod(**inputs)
2025-09-07T08:06:33.3202944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3203441Z     outputs = self.bert(
2025-09-07T08:06:33.3203696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3203762Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3204124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3204192Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3204412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3204486Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3204729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3204805Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3205045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3205107Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3205348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3205500Z     self_outputs = self.self(
2025-09-07T08:06:33.3205753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3205817Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3206076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3206139Z     self.key(current_states)
2025-09-07T08:06:33.3206143Z 
2025-09-07T08:06:33.3206238Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3206438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3206497Z     return mod(**inputs)
2025-09-07T08:06:33.3206747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3206801Z     outputs = self.bert(
2025-09-07T08:06:33.3207049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3207121Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3207361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3207424Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3207639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3207713Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3207953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3208024Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3208265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3208324Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3208573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3208632Z     self_outputs = self.self(
2025-09-07T08:06:33.3208867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3208928Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3209172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3209241Z     self.value(current_states)
2025-09-07T08:06:33.3209244Z 
2025-09-07T08:06:33.3209319Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3209409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3209603Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3209660Z     return mod(**inputs)
2025-09-07T08:06:33.3209981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3210037Z     outputs = self.bert(
2025-09-07T08:06:33.3210282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3210352Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3210598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3210662Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3210875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3210949Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3211197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3211330Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3211574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3211633Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3211880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3211944Z     self_outputs = self.self(
2025-09-07T08:06:33.3212179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3212243Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3212485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3212616Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3212620Z 
2025-09-07T08:06:33.3212716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3212916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3212976Z     return mod(**inputs)
2025-09-07T08:06:33.3213229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3213288Z     outputs = self.bert(
2025-09-07T08:06:33.3213531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3213601Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3213842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3213905Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3214123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3214196Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3214444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3214511Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3214746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3214813Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3215054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3215174Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3215414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3215496Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3215499Z 
2025-09-07T08:06:33.3215657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3215843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3215903Z     return mod(**inputs)
2025-09-07T08:06:33.3216152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3216214Z     outputs = self.bert(
2025-09-07T08:06:33.3216459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3216521Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3216767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3216829Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3217044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3217181Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3217432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3217560Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3217825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3217897Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3218175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3218292Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3218536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3218610Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3218617Z 
2025-09-07T08:06:33.3218713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3218900Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3218961Z     return mod(**inputs)
2025-09-07T08:06:33.3219207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3219261Z     outputs = self.bert(
2025-09-07T08:06:33.3219508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3219570Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3219816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3219875Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3220092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3220166Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3220406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3220482Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3220739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3220810Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3221083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3221190Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3221432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3221535Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3221824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3221886Z     return self.act(input)
2025-09-07T08:06:33.3221891Z 
2025-09-07T08:06:33.3221986Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3222171Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3222228Z     return mod(**inputs)
2025-09-07T08:06:33.3222472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3222529Z     outputs = self.bert(
2025-09-07T08:06:33.3222777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3222844Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3223139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3223208Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3223423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3223491Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3223729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3223801Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3224064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3224129Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3224406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3224535Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3224788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3224860Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3224863Z 
2025-09-07T08:06:33.3224952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3225143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3225202Z     return mod(**inputs)
2025-09-07T08:06:33.3225457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3225516Z     outputs = self.bert(
2025-09-07T08:06:33.3225758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3225829Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3226074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3226141Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3226354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3226423Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3226679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3226751Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3226998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3227058Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3227309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3227435Z     self_outputs = self.self(
2025-09-07T08:06:33.3227676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3227745Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3227985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3228196Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3228200Z 
2025-09-07T08:06:33.3228289Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3228479Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3228544Z     return mod(**inputs)
2025-09-07T08:06:33.3228792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3228856Z     outputs = self.bert(
2025-09-07T08:06:33.3229157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3229233Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3229479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3229543Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3229772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3229843Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3230090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3230160Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3230401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3230477Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3230719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3230790Z     self_outputs = self.self(
2025-09-07T08:06:33.3231028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3231092Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3231342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3231406Z     self.key(current_states)
2025-09-07T08:06:33.3231409Z 
2025-09-07T08:06:33.3231512Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3231700Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3231765Z     return mod(**inputs)
2025-09-07T08:06:33.3232018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3232077Z     outputs = self.bert(
2025-09-07T08:06:33.3232329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3232395Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3232641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3232703Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3232919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3232994Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3233235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3233375Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3233620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3233682Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3233932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3233996Z     self_outputs = self.self(
2025-09-07T08:06:33.3234239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3234299Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3234539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3234611Z     self.value(current_states)
2025-09-07T08:06:33.3234614Z 
2025-09-07T08:06:33.3234690Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3234851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3235040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3235108Z     return mod(**inputs)
2025-09-07T08:06:33.3235355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3235416Z     outputs = self.bert(
2025-09-07T08:06:33.3235666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3235730Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3235976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3236039Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3236252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3236333Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3236574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3236652Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3236891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3236951Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3237200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3237260Z     self_outputs = self.self(
2025-09-07T08:06:33.3237502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3237562Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3237812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3237939Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3237943Z 
2025-09-07T08:06:33.3238033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3238226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3238281Z     return mod(**inputs)
2025-09-07T08:06:33.3238533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3238588Z     outputs = self.bert(
2025-09-07T08:06:33.3238835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3238904Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3239147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3239304Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3239521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3239589Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3239837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3239911Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3240155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3240218Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3240466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3240586Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3240942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3241030Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3241033Z 
2025-09-07T08:06:33.3241125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3241318Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3241375Z     return mod(**inputs)
2025-09-07T08:06:33.3241623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3241688Z     outputs = self.bert(
2025-09-07T08:06:33.3241932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3242003Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3242244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3242318Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3242532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3242599Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3242847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3242919Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3243183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3243252Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3243532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3243649Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3243900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3243980Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3243983Z 
2025-09-07T08:06:33.3244076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3244265Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3244322Z     return mod(**inputs)
2025-09-07T08:06:33.3244566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3244630Z     outputs = self.bert(
2025-09-07T08:06:33.3244876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3244944Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3245186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3245311Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3245536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3245605Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3245850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3245922Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3246178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3246249Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3246522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3246653Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3246963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3247074Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3247286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3247352Z     return self.act(input)
2025-09-07T08:06:33.3247355Z 
2025-09-07T08:06:33.3247454Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3247637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3247699Z     return mod(**inputs)
2025-09-07T08:06:33.3247945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3248002Z     outputs = self.bert(
2025-09-07T08:06:33.3248252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3248319Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3248561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3248620Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3248829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3248896Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3249135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3249211Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3249473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3249543Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3249823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3249949Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3250200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3250274Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3250277Z 
2025-09-07T08:06:33.3250374Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3250565Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3250622Z     return mod(**inputs)
2025-09-07T08:06:33.3250873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3250930Z     outputs = self.bert(
2025-09-07T08:06:33.3251239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3251302Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3251546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3251606Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3251818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3251891Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3252130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3252202Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3252438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3252560Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3252804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3252868Z     self_outputs = self.self(
2025-09-07T08:06:33.3253104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3253163Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3253401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3253606Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3253611Z 
2025-09-07T08:06:33.3253702Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3253887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3253945Z     return mod(**inputs)
2025-09-07T08:06:33.3254197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3254255Z     outputs = self.bert(
2025-09-07T08:06:33.3254498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3254568Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3254810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3254880Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3255092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3255158Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3255406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3255482Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3255724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3255784Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3256026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3256094Z     self_outputs = self.self(
2025-09-07T08:06:33.3256332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3256397Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3256639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3256707Z     self.key(current_states)
2025-09-07T08:06:33.3256710Z 
2025-09-07T08:06:33.3256801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3257058Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3257129Z     return mod(**inputs)
2025-09-07T08:06:33.3257379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3257441Z     outputs = self.bert(
2025-09-07T08:06:33.3257739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3257811Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3258066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3258131Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3258356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3258501Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3258750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3258832Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3259075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3259146Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3259389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3259459Z     self_outputs = self.self(
2025-09-07T08:06:33.3259696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3259759Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3260013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3260083Z     self.value(current_states)
2025-09-07T08:06:33.3260087Z 
2025-09-07T08:06:33.3260167Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3260261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3260448Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3260513Z     return mod(**inputs)
2025-09-07T08:06:33.3260761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3260831Z     outputs = self.bert(
2025-09-07T08:06:33.3261077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3261143Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3261390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3261460Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3261684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3261754Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3262000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3262070Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3262307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3262375Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3262614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3262683Z     self_outputs = self.self(
2025-09-07T08:06:33.3262920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3263060Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3263311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3263440Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3263444Z 
2025-09-07T08:06:33.3263542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3263726Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3263787Z     return mod(**inputs)
2025-09-07T08:06:33.3264037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3264098Z     outputs = self.bert(
2025-09-07T08:06:33.3264353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3264482Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3264743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3264811Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3265027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3265110Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3265353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3265435Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3265673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3265732Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3265988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3266116Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3266365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3266438Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3266441Z 
2025-09-07T08:06:33.3266539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3266724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3266829Z     return mod(**inputs)
2025-09-07T08:06:33.3267244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3267332Z     outputs = self.bert(
2025-09-07T08:06:33.3267657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3267756Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3268029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3268182Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3268439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3268563Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3268851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3268952Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3269258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3269393Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3269820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3269969Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3270313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3270413Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3270417Z 
2025-09-07T08:06:33.3270520Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3270817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3270917Z     return mod(**inputs)
2025-09-07T08:06:33.3271226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3271311Z     outputs = self.bert(
2025-09-07T08:06:33.3271641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3271793Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3272098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3272225Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3272475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3272599Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3272858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3273010Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3273341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3273434Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3273773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3273905Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3274248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3274396Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3274640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3274801Z     return self.act(input)
2025-09-07T08:06:33.3274805Z 
2025-09-07T08:06:33.3274922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3275166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3275296Z     return mod(**inputs)
2025-09-07T08:06:33.3275592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3275710Z     outputs = self.bert(
2025-09-07T08:06:33.3275984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3276118Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3276379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3276566Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3276813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3276909Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3277210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3277386Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3277736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3277844Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3278145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3278342Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3278612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3278730Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3278733Z 
2025-09-07T08:06:33.3278892Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3279221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3279308Z     return mod(**inputs)
2025-09-07T08:06:33.3279653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3279783Z     outputs = self.bert(
2025-09-07T08:06:33.3280055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3280229Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3280514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3280634Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3280875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3280970Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3281294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3281414Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3281736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3281825Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3282095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3282204Z     self_outputs = self.self(
2025-09-07T08:06:33.3282510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3282656Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3282926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3283186Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3283192Z 
2025-09-07T08:06:33.3283313Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3283514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3283740Z     return mod(**inputs)
2025-09-07T08:06:33.3284018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3284130Z     outputs = self.bert(
2025-09-07T08:06:33.3284403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3284496Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3284828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3284945Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3285220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3285381Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3285684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3285784Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3286097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3286230Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3286504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3286623Z     self_outputs = self.self(
2025-09-07T08:06:33.3286902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3286980Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3287456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3287552Z     self.key(current_states)
2025-09-07T08:06:33.3287555Z 
2025-09-07T08:06:33.3287706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3287988Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3288106Z     return mod(**inputs)
2025-09-07T08:06:33.3288425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3288523Z     outputs = self.bert(
2025-09-07T08:06:33.3288830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3288937Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3289248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3289333Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3289624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3289767Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3290042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3290185Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3290459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3290607Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3290890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3290981Z     self_outputs = self.self(
2025-09-07T08:06:33.3291297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3291390Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3291679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3291834Z     self.value(current_states)
2025-09-07T08:06:33.3291838Z 
2025-09-07T08:06:33.3291980Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3292121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3292342Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3292511Z     return mod(**inputs)
2025-09-07T08:06:33.3292778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3292945Z     outputs = self.bert(
2025-09-07T08:06:33.3293235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3293397Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3293703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3293798Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3294099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3294221Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3294494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3294623Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3294891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3294997Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3295382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3295521Z     self_outputs = self.self(
2025-09-07T08:06:33.3295784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3295874Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3296173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3296329Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3296333Z 
2025-09-07T08:06:33.3296536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3296808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3296893Z     return mod(**inputs)
2025-09-07T08:06:33.3297208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3297309Z     outputs = self.bert(
2025-09-07T08:06:33.3297695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3297800Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3298111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3298202Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3298448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3298576Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3298883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3299027Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3299296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3299386Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3299703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3299837Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3300186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3300292Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3300296Z 
2025-09-07T08:06:33.3300452Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3300681Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3300765Z     return mod(**inputs)
2025-09-07T08:06:33.3301103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3301312Z     outputs = self.bert(
2025-09-07T08:06:33.3301633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3301727Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3302028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3302106Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3302390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3302544Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3302833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3302961Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3303472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3303559Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3303967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3304114Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3304423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3304527Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3304531Z 
2025-09-07T08:06:33.3304673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3304934Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3305043Z     return mod(**inputs)
2025-09-07T08:06:33.3305365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3305452Z     outputs = self.bert(
2025-09-07T08:06:33.3305758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3305840Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3306264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3306361Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3306610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3306737Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3307016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3307218Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3307527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3307625Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3307964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3308106Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3308396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3308579Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3308862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3308953Z     return self.act(input)
2025-09-07T08:06:33.3308957Z 
2025-09-07T08:06:33.3309083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3309447Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3309523Z     return mod(**inputs)
2025-09-07T08:06:33.3309901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3309994Z     outputs = self.bert(
2025-09-07T08:06:33.3310272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3310409Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3310731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3310877Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3311138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3311324Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3311618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3311724Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3312035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3312167Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3312518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3312683Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3312956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3313086Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3313092Z 
2025-09-07T08:06:33.3313204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3313503Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3313588Z     return mod(**inputs)
2025-09-07T08:06:33.3313905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3313991Z     outputs = self.bert(
2025-09-07T08:06:33.3314263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3314409Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3314694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3314827Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3315123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3315254Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3315512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3315648Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3315984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3316080Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3316377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3316470Z     self_outputs = self.self(
2025-09-07T08:06:33.3316723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3316909Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3317184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3317514Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3317517Z 
2025-09-07T08:06:33.3317643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3317880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3318015Z     return mod(**inputs)
2025-09-07T08:06:33.3318309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3318427Z     outputs = self.bert(
2025-09-07T08:06:33.3318702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3318823Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3319159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3319292Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3319622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3319722Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3320024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3320141Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3320469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3320574Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3320844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3320966Z     self_outputs = self.self(
2025-09-07T08:06:33.3321249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3321357Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3321665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3321768Z     self.key(current_states)
2025-09-07T08:06:33.3321803Z 
2025-09-07T08:06:33.3321926Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3322143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3322273Z     return mod(**inputs)
2025-09-07T08:06:33.3322539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3322708Z     outputs = self.bert(
2025-09-07T08:06:33.3322986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3323084Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3323400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3323494Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3323798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3323948Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3324219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3324361Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3324628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3324735Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3325034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3325238Z     self_outputs = self.self(
2025-09-07T08:06:33.3325505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3325590Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3325891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3331907Z     self.value(current_states)
2025-09-07T08:06:33.3331917Z 
2025-09-07T08:06:33.3332019Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3332130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3332344Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3332406Z     return mod(**inputs)
2025-09-07T08:06:33.3332797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3332866Z     outputs = self.bert(
2025-09-07T08:06:33.3333139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3333207Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3333459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3333524Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3333743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3333821Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3334069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3334146Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3334409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3334472Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3334720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3334783Z     self_outputs = self.self(
2025-09-07T08:06:33.3335021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3335079Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3335317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3335450Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3335454Z 
2025-09-07T08:06:33.3335557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3335771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3335832Z     return mod(**inputs)
2025-09-07T08:06:33.3336085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3336142Z     outputs = self.bert(
2025-09-07T08:06:33.3336385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3336453Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3336691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3336752Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3336979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3337048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3337657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3337731Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3337967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3338027Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3338270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3338393Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3338634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3338713Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3338717Z 
2025-09-07T08:06:33.3338813Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3339082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3339148Z     return mod(**inputs)
2025-09-07T08:06:33.3339393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3339455Z     outputs = self.bert(
2025-09-07T08:06:33.3339709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3339778Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3340036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3340100Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3340327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3340395Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3340646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3340718Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3340983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3341055Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3341334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3341453Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3341702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3341781Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3341789Z 
2025-09-07T08:06:33.3341887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3342090Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3342153Z     return mod(**inputs)
2025-09-07T08:06:33.3342405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3342468Z     outputs = self.bert(
2025-09-07T08:06:33.3342712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3342779Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3343027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3343092Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3343312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3343446Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3343692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3343769Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3344029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3344099Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3344376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3344490Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3344733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3344835Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3345106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3345171Z     return self.act(input)
2025-09-07T08:06:33.3345175Z 
2025-09-07T08:06:33.3345272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3345465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3345520Z     return mod(**inputs)
2025-09-07T08:06:33.3345770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3345826Z     outputs = self.bert(
2025-09-07T08:06:33.3346081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3346146Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3346384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3346456Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3346681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3346751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3346995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3347070Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3347326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3347391Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3347666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3347790Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3348046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3348117Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3348120Z 
2025-09-07T08:06:33.3348212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3348404Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3348459Z     return mod(**inputs)
2025-09-07T08:06:33.3348709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3348764Z     outputs = self.bert(
2025-09-07T08:06:33.3349009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3349073Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3349314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3349447Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3349663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3349736Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3349992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3350064Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3350313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3350374Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3350618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3350680Z     self_outputs = self.self(
2025-09-07T08:06:33.3350975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3351043Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3351281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:06:33.3351494Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:06:33.3351498Z 
2025-09-07T08:06:33.3351586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3351777Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3351834Z     return mod(**inputs)
2025-09-07T08:06:33.3352085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3352146Z     outputs = self.bert(
2025-09-07T08:06:33.3352391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3352462Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3352701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3352761Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3352980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3353045Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3353289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3353357Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3353589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3353654Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3353899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3353963Z     self_outputs = self.self(
2025-09-07T08:06:33.3354198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3354262Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3354502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:06:33.3354562Z     self.key(current_states)
2025-09-07T08:06:33.3354565Z 
2025-09-07T08:06:33.3354658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3354841Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3354900Z     return mod(**inputs)
2025-09-07T08:06:33.3355147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3355268Z     outputs = self.bert(
2025-09-07T08:06:33.3355515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3355577Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3355822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3355886Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3356101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3356174Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3356414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3356490Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3356781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3356848Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3357089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3357152Z     self_outputs = self.self(
2025-09-07T08:06:33.3357396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3357456Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3357704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:06:33.3357767Z     self.value(current_states)
2025-09-07T08:06:33.3357770Z 
2025-09-07T08:06:33.3357844Z cudagraph partition due to non gpu ops
2025-09-07T08:06:33.3357941Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3358132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3358190Z     return mod(**inputs)
2025-09-07T08:06:33.3358433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3358490Z     outputs = self.bert(
2025-09-07T08:06:33.3358737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3358800Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3359045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3359106Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3359322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3359386Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3359632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3359710Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3359946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3360007Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3360249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:06:33.3360309Z     self_outputs = self.self(
2025-09-07T08:06:33.3360548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3360605Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3360845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:06:33.3361053Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:06:33.3361057Z 
2025-09-07T08:06:33.3361153Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3361338Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3361393Z     return mod(**inputs)
2025-09-07T08:06:33.3361641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3361698Z     outputs = self.bert(
2025-09-07T08:06:33.3361946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3362009Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3362250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3362318Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3362592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3362664Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3362901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:06:33.3362970Z     self_attention_outputs = self.attention(
2025-09-07T08:06:33.3363212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:06:33.3363271Z     return func(*args, **kwargs)
2025-09-07T08:06:33.3363511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:06:33.3363628Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:06:33.3363868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:06:33.3363951Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3363954Z 
2025-09-07T08:06:33.3364043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3364237Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3364293Z     return mod(**inputs)
2025-09-07T08:06:33.3364539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3364594Z     outputs = self.bert(
2025-09-07T08:06:33.3364836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3364897Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3365135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3365194Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3365410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3365478Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3365725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3365797Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3366065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3366132Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3366416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3366527Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3366774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:06:33.3366915Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3366918Z 
2025-09-07T08:06:33.3367011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3367199Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3367254Z     return mod(**inputs)
2025-09-07T08:06:33.3367499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3367561Z     outputs = self.bert(
2025-09-07T08:06:33.3367802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3367870Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3368110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3368231Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3368455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3368520Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3368769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3368839Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3369100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3369163Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3369433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:06:33.3369537Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:06:33.3369782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:06:33.3369885Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:06:33.3370092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:06:33.3370157Z     return self.act(input)
2025-09-07T08:06:33.3370162Z 
2025-09-07T08:06:33.3370252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3370435Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3370492Z     return mod(**inputs)
2025-09-07T08:06:33.3370735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:06:33.3370791Z     outputs = self.bert(
2025-09-07T08:06:33.3371032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:06:33.3371097Z     encoder_outputs = self.encoder(
2025-09-07T08:06:33.3371337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:06:33.3371395Z     layer_outputs = layer_module(
2025-09-07T08:06:33.3371607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:06:33.3371683Z     return super().__call__(*args, **kwargs)
2025-09-07T08:06:33.3371928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:06:33.3372000Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:06:33.3372253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:06:33.3372317Z     return forward_fn(*input_tensors)
2025-09-07T08:06:33.3372588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:06:33.3372775Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:06:33.3373020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:06:33.3373090Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3373093Z 
2025-09-07T08:06:33.3373189Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3373376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3373438Z     return mod(**inputs)
2025-09-07T08:06:33.3373684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T08:06:33.3373770Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:06:33.3374072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T08:06:33.3374182Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:06:33.3374432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward
2025-09-07T08:06:33.3374513Z     hidden_states = self.transform(hidden_states)
2025-09-07T08:06:33.3374754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward
2025-09-07T08:06:33.3374831Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:06:33.3374834Z 
2025-09-07T08:06:33.3374923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:06:33.3375111Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:06:33.3375169Z     return mod(**inputs)
2025-09-07T08:06:33.3375422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T08:06:33.3375505Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:06:33.3375746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T08:06:33.3375843Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:06:33.3376086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward
2025-09-07T08:06:33.3376171Z     hidden_states = self.decoder(hidden_states)
2025-09-07T08:06:33.3376175Z 
2025-09-07T08:06:41.7343677Z pass
2025-09-07T08:06:41.7344063Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:44.3344452Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:06:44.3345457Z   import pynvml  # type: ignore[import]
2025-09-07T08:06:46.3398394Z 
2025-09-07T08:06:51.7048172Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:06:51.7048445Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:06:51.7441845Z cpu  eval  hf_Bert_large                      
2025-09-07T08:06:54.0732257Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:54.7221263Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:06:55.3606887Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:07:07.1989380Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.1989846Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.1990613Z     return mod(**inputs)
2025-09-07T08:07:07.1991028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.1991438Z     outputs = self.bert(
2025-09-07T08:07:07.1991813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 964, in forward
2025-09-07T08:07:07.1992239Z     embedding_output = self.embeddings(
2025-09-07T08:07:07.1992638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 188, in forward
2025-09-07T08:07:07.1993029Z     embeddings = self.LayerNorm(embeddings)
2025-09-07T08:07:07.1993165Z 
2025-09-07T08:07:07.1993244Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1993454Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1993647Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1993841Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1994036Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1994377Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1994577Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1994774Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1994980Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1995167Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1995361Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1995593Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1995815Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1996002Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1996191Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1996380Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1996570Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1996754Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1996943Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1997145Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1997338Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1997523Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1997715Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1997904Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.1998140Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.1998516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.1998846Z     return mod(**inputs)
2025-09-07T08:07:07.1999212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.1999592Z     outputs = self.bert(
2025-09-07T08:07:07.1999943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2000390Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2000780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2001212Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2001584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2001997Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2002404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2002802Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2003323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2003705Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2004076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2004592Z     self_outputs = self.self(
2025-09-07T08:07:07.2004964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2005339Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2005712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2006239Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2006511Z 
2025-09-07T08:07:07.2006619Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2006991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2007322Z     return mod(**inputs)
2025-09-07T08:07:07.2007686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2008058Z     outputs = self.bert(
2025-09-07T08:07:07.2008515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2008915Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2009287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2009663Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2010017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2010388Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2010767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2011149Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2011520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2011907Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2012269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2012645Z     self_outputs = self.self(
2025-09-07T08:07:07.2013000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2013392Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2013779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2014152Z     self.key(current_states)
2025-09-07T08:07:07.2014259Z 
2025-09-07T08:07:07.2014370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2014714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2015039Z     return mod(**inputs)
2025-09-07T08:07:07.2015397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2015768Z     outputs = self.bert(
2025-09-07T08:07:07.2016116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2016499Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2016872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2017250Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2017646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2018000Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2018384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2018850Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2019240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2019625Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2019990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2020366Z     self_outputs = self.self(
2025-09-07T08:07:07.2020714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2021088Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2021457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2021831Z     self.value(current_states)
2025-09-07T08:07:07.2021951Z 
2025-09-07T08:07:07.2022026Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2022258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2022681Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2023000Z     return mod(**inputs)
2025-09-07T08:07:07.2023363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2023749Z     outputs = self.bert(
2025-09-07T08:07:07.2024106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2024508Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2024881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2025267Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2025620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2025996Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2026377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2026759Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2027141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2027512Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2027876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2028247Z     self_outputs = self.self(
2025-09-07T08:07:07.2028599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2028963Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2029322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2029762Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2029954Z 
2025-09-07T08:07:07.2030055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2030432Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2030760Z     return mod(**inputs)
2025-09-07T08:07:07.2031125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2031514Z     outputs = self.bert(
2025-09-07T08:07:07.2031856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2032247Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2032616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2033110Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2033443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2033809Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2034189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2034579Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2034960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2035322Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2035726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2036169Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2036703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2037109Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2037245Z 
2025-09-07T08:07:07.2037345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2037705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2038025Z     return mod(**inputs)
2025-09-07T08:07:07.2038388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2038764Z     outputs = self.bert(
2025-09-07T08:07:07.2039109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2039493Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2039871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2040261Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2040596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2040953Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2041342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2041761Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2042170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2042558Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2042978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2043478Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2043927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2044310Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2044451Z 
2025-09-07T08:07:07.2044554Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2044908Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2045231Z     return mod(**inputs)
2025-09-07T08:07:07.2045580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2045951Z     outputs = self.bert(
2025-09-07T08:07:07.2046303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2046686Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2047058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2047511Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2047856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2048219Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2048603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2048985Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2049373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2049786Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2050192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2050650Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2051133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2051555Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2051942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2052282Z     return self.act(input)
2025-09-07T08:07:07.2052393Z 
2025-09-07T08:07:07.2052500Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2052858Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2053169Z     return mod(**inputs)
2025-09-07T08:07:07.2053517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2053901Z     outputs = self.bert(
2025-09-07T08:07:07.2054256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2054631Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2055005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2055387Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2055729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2056085Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2056452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2056843Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2057237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2057694Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2058110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2058568Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2059022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2059413Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2059552Z 
2025-09-07T08:07:07.2059654Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2060006Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2060327Z     return mod(**inputs)
2025-09-07T08:07:07.2060695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2061099Z     outputs = self.bert(
2025-09-07T08:07:07.2061533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2061909Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2062279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2062657Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2062998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2063352Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2063724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2064105Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2064492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2064935Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2065300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2065670Z     self_outputs = self.self(
2025-09-07T08:07:07.2066033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2066403Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2066762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2067277Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2067546Z 
2025-09-07T08:07:07.2067640Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2067991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2068314Z     return mod(**inputs)
2025-09-07T08:07:07.2068667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2069035Z     outputs = self.bert(
2025-09-07T08:07:07.2069375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2069755Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2070121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2070492Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2070819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2071171Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2071544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2071948Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2072310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2072677Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2073033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2073427Z     self_outputs = self.self(
2025-09-07T08:07:07.2073777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2074137Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2074489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2074864Z     self.key(current_states)
2025-09-07T08:07:07.2074974Z 
2025-09-07T08:07:07.2075078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2075506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2075816Z     return mod(**inputs)
2025-09-07T08:07:07.2076162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2076535Z     outputs = self.bert(
2025-09-07T08:07:07.2076881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2077254Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2077621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2077994Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2078327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2078742Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2079108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2079487Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2079857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2080223Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2080569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2080939Z     self_outputs = self.self(
2025-09-07T08:07:07.2081285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2081646Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2081998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2082371Z     self.value(current_states)
2025-09-07T08:07:07.2082490Z 
2025-09-07T08:07:07.2082566Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2082789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2083144Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2083458Z     return mod(**inputs)
2025-09-07T08:07:07.2083803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2084176Z     outputs = self.bert(
2025-09-07T08:07:07.2084522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2084913Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2085268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2085652Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2085982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2086330Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2086696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2087076Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2087443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2087806Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2088159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2088524Z     self_outputs = self.self(
2025-09-07T08:07:07.2088872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2090440Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2090801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2091244Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2091426Z 
2025-09-07T08:07:07.2091521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2091873Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2092188Z     return mod(**inputs)
2025-09-07T08:07:07.2092537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2092905Z     outputs = self.bert(
2025-09-07T08:07:07.2093245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2093692Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2094058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2094434Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2094759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2095114Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2095484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2095863Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2096237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2096602Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2096959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2097390Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2097873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2098252Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2098391Z 
2025-09-07T08:07:07.2098482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2098823Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2099138Z     return mod(**inputs)
2025-09-07T08:07:07.2099480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2099848Z     outputs = self.bert(
2025-09-07T08:07:07.2100189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2100570Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2100933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2101299Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2101635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2101983Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2102357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2102741Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2103317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2103719Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2104248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2104707Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2105130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2105507Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2105643Z 
2025-09-07T08:07:07.2105737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2106083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2106396Z     return mod(**inputs)
2025-09-07T08:07:07.2106735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2107111Z     outputs = self.bert(
2025-09-07T08:07:07.2107561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2107966Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2108346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2108725Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2109064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2109423Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2109803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2110191Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2110589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2110983Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2111398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2111857Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2112268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2112686Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2113064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2113406Z     return self.act(input)
2025-09-07T08:07:07.2113957Z 
2025-09-07T08:07:07.2114064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2114412Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2114733Z     return mod(**inputs)
2025-09-07T08:07:07.2115094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2115473Z     outputs = self.bert(
2025-09-07T08:07:07.2115825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2116199Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2116567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2116947Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2117290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2117640Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2118016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2118401Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2118889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2119285Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2119682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2120142Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2120578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2120963Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2121095Z 
2025-09-07T08:07:07.2121198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2121540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2121864Z     return mod(**inputs)
2025-09-07T08:07:07.2122277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2122657Z     outputs = self.bert(
2025-09-07T08:07:07.2122997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2123376Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2123742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2124117Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2124454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2124807Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2125194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2125585Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2125962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2126331Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2126686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2127060Z     self_outputs = self.self(
2025-09-07T08:07:07.2127412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2127786Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2128134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2128670Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2128942Z 
2025-09-07T08:07:07.2129047Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2129401Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2129720Z     return mod(**inputs)
2025-09-07T08:07:07.2130078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2130453Z     outputs = self.bert(
2025-09-07T08:07:07.2130801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2131181Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2131547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2131915Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2132252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2132692Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2133080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2133465Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2133847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2134217Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2134585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2134959Z     self_outputs = self.self(
2025-09-07T08:07:07.2135307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2135670Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2136099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2136482Z     self.key(current_states)
2025-09-07T08:07:07.2136591Z 
2025-09-07T08:07:07.2136698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2137044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2137362Z     return mod(**inputs)
2025-09-07T08:07:07.2137795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2138182Z     outputs = self.bert(
2025-09-07T08:07:07.2138525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2138907Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2139278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2139659Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2139996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2140350Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2140725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2141109Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2141479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2141848Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2142206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2142579Z     self_outputs = self.self(
2025-09-07T08:07:07.2142934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2143302Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2143647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2144015Z     self.value(current_states)
2025-09-07T08:07:07.2144129Z 
2025-09-07T08:07:07.2144205Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2144433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2144778Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2145097Z     return mod(**inputs)
2025-09-07T08:07:07.2145450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2145831Z     outputs = self.bert(
2025-09-07T08:07:07.2146173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2146639Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2147005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2147382Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2147716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2148063Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2148438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2148819Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2149190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2149553Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2149963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2150350Z     self_outputs = self.self(
2025-09-07T08:07:07.2150706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2151077Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2151439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2151866Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2152057Z 
2025-09-07T08:07:07.2152156Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2152509Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2152831Z     return mod(**inputs)
2025-09-07T08:07:07.2153182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2153656Z     outputs = self.bert(
2025-09-07T08:07:07.2154002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2154382Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2154749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2155118Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2155459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2155816Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2156192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2156576Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2156948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2157318Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2157676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2158107Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2158523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2158911Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2159054Z 
2025-09-07T08:07:07.2159148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2159498Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2159814Z     return mod(**inputs)
2025-09-07T08:07:07.2160162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2160614Z     outputs = self.bert(
2025-09-07T08:07:07.2160960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2161340Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2161699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2162077Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2162414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2162772Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2163377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2163881Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2164438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2176456Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2176988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2177496Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2178025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2178424Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2178571Z 
2025-09-07T08:07:07.2178676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2179045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2179377Z     return mod(**inputs)
2025-09-07T08:07:07.2179750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2180139Z     outputs = self.bert(
2025-09-07T08:07:07.2180490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2180873Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2181257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2181633Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2181979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2182345Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2182730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2183120Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2183522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2183919Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2184329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2184790Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2185208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2185624Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2186007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2186374Z     return self.act(input)
2025-09-07T08:07:07.2186482Z 
2025-09-07T08:07:07.2186592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2187067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2187390Z     return mod(**inputs)
2025-09-07T08:07:07.2187754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2188135Z     outputs = self.bert(
2025-09-07T08:07:07.2188495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2188874Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2189252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2189636Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2189981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2190340Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2190791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2191190Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2191625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2192025Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2192430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2192901Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2193344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2193735Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2193876Z 
2025-09-07T08:07:07.2193989Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2194342Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2194666Z     return mod(**inputs)
2025-09-07T08:07:07.2195024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2195403Z     outputs = self.bert(
2025-09-07T08:07:07.2195740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2196121Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2196488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2196863Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2197204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2197578Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2197955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2198354Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2198735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2199103Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2199462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2199834Z     self_outputs = self.self(
2025-09-07T08:07:07.2200194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2200562Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2200921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2201520Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2201791Z 
2025-09-07T08:07:07.2201890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2202248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2202570Z     return mod(**inputs)
2025-09-07T08:07:07.2202919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2203474Z     outputs = self.bert(
2025-09-07T08:07:07.2203827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2204211Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2204699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2205095Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2205438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2205797Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2206178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2206560Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2206934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2207304Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2207664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2208038Z     self_outputs = self.self(
2025-09-07T08:07:07.2208391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2208753Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2209111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2209486Z     self.key(current_states)
2025-09-07T08:07:07.2209595Z 
2025-09-07T08:07:07.2209695Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2210052Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2210368Z     return mod(**inputs)
2025-09-07T08:07:07.2210718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2211091Z     outputs = self.bert(
2025-09-07T08:07:07.2211433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2211816Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2212189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2212568Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2212905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2213262Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2213643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2214028Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2214400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2214763Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2215121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2215643Z     self_outputs = self.self(
2025-09-07T08:07:07.2215997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2216366Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2216716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2217089Z     self.value(current_states)
2025-09-07T08:07:07.2217200Z 
2025-09-07T08:07:07.2217284Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2217563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2217913Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2218229Z     return mod(**inputs)
2025-09-07T08:07:07.2218646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2219026Z     outputs = self.bert(
2025-09-07T08:07:07.2219371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2219744Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2220110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2220487Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2220824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2221190Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2221567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2221949Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2222333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2222702Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2223048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2223418Z     self_outputs = self.self(
2025-09-07T08:07:07.2223764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2224130Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2224478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2224912Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2225101Z 
2025-09-07T08:07:07.2225198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2225547Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2225866Z     return mod(**inputs)
2025-09-07T08:07:07.2226206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2226580Z     outputs = self.bert(
2025-09-07T08:07:07.2226926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2227309Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2227675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2228041Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2228379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2228754Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2229213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2229591Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2229974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2230343Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2230694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2231120Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2231533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2231911Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2232042Z 
2025-09-07T08:07:07.2232138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2232560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2232871Z     return mod(**inputs)
2025-09-07T08:07:07.2233209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2233574Z     outputs = self.bert(
2025-09-07T08:07:07.2233906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2234287Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2234643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2235004Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2235325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2235662Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2236027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2236402Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2236792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2237177Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2237572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2238019Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2238441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2238834Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2238963Z 
2025-09-07T08:07:07.2239061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2239412Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2239716Z     return mod(**inputs)
2025-09-07T08:07:07.2240060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2240427Z     outputs = self.bert(
2025-09-07T08:07:07.2240758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2241126Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2241484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2241848Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2242175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2242518Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2242958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2243331Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2243717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2244097Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2244492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2244932Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2245340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2245744Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2246178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2246514Z     return self.act(input)
2025-09-07T08:07:07.2246619Z 
2025-09-07T08:07:07.2246713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2247063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2247378Z     return mod(**inputs)
2025-09-07T08:07:07.2247729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2248101Z     outputs = self.bert(
2025-09-07T08:07:07.2248434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2248806Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2249158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2249525Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2249848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2250190Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2250554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2250924Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2251309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2251688Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2252078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2252529Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2252958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2253330Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2253459Z 
2025-09-07T08:07:07.2253550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2253888Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2254194Z     return mod(**inputs)
2025-09-07T08:07:07.2254525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2254887Z     outputs = self.bert(
2025-09-07T08:07:07.2255218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2255588Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2255942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2256402Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2256732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2257078Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2257438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2257868Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2258233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2258592Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2258941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2259302Z     self_outputs = self.self(
2025-09-07T08:07:07.2259707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2260074Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2260419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2260925Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2261185Z 
2025-09-07T08:07:07.2261276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2261615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2261921Z     return mod(**inputs)
2025-09-07T08:07:07.2262256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2262617Z     outputs = self.bert(
2025-09-07T08:07:07.2262947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2263316Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2263667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2264035Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2264363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2264705Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2265065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2265440Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2265803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2266159Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2266507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2266868Z     self_outputs = self.self(
2025-09-07T08:07:07.2267204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2267557Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2267895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2268258Z     self.key(current_states)
2025-09-07T08:07:07.2268367Z 
2025-09-07T08:07:07.2268463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2268808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2269132Z     return mod(**inputs)
2025-09-07T08:07:07.2269481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2269931Z     outputs = self.bert(
2025-09-07T08:07:07.2270267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2270639Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2270997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2271363Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2271693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2272040Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2272404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2272778Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2273426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2273795Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2274144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2274506Z     self_outputs = self.self(
2025-09-07T08:07:07.2274847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2275199Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2275543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2275908Z     self.value(current_states)
2025-09-07T08:07:07.2276014Z 
2025-09-07T08:07:07.2276088Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2276306Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2276652Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2276959Z     return mod(**inputs)
2025-09-07T08:07:07.2277297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2277663Z     outputs = self.bert(
2025-09-07T08:07:07.2277995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2278362Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2278716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2279082Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2279409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2279756Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2280125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2280493Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2280859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2281218Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2281571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2281933Z     self_outputs = self.self(
2025-09-07T08:07:07.2282274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2282634Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2282977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2283498Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2283680Z 
2025-09-07T08:07:07.2283773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2284109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2284412Z     return mod(**inputs)
2025-09-07T08:07:07.2284748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2285111Z     outputs = self.bert(
2025-09-07T08:07:07.2285442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2285808Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2286158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2286520Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2286917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2287265Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2287629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2288002Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2288359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2288719Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2289071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2289487Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2289901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2290297Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2290431Z 
2025-09-07T08:07:07.2290526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2290862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2291168Z     return mod(**inputs)
2025-09-07T08:07:07.2291502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2291864Z     outputs = self.bert(
2025-09-07T08:07:07.2292197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2292564Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2292918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2293282Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2293608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2293948Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2294307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2294685Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2295075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2295456Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2295852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2296295Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2296711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2297224Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2297383Z 
2025-09-07T08:07:07.2297509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2297924Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2298246Z     return mod(**inputs)
2025-09-07T08:07:07.2298586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2298950Z     outputs = self.bert(
2025-09-07T08:07:07.2299304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2299671Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2300029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2300462Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2300785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2301146Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2301524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2301897Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2302279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2302669Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2303200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2303664Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2304080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2304487Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2304862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2305197Z     return self.act(input)
2025-09-07T08:07:07.2305301Z 
2025-09-07T08:07:07.2305396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2305751Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2306056Z     return mod(**inputs)
2025-09-07T08:07:07.2306413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2306795Z     outputs = self.bert(
2025-09-07T08:07:07.2307134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2307532Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2307883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2308256Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2308584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2308928Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2309290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2309662Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2310053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2315013Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2315443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2315996Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2316430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2316831Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2316967Z 
2025-09-07T08:07:07.2317085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2317440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2317766Z     return mod(**inputs)
2025-09-07T08:07:07.2318125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2318496Z     outputs = self.bert(
2025-09-07T08:07:07.2318953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2319349Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2319722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2320099Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2320435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2320803Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2321205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2321585Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2321984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2322353Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2322726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2323103Z     self_outputs = self.self(
2025-09-07T08:07:07.2323454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2323819Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2324185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2324712Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2324975Z 
2025-09-07T08:07:07.2325075Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2325454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2325785Z     return mod(**inputs)
2025-09-07T08:07:07.2326138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2326519Z     outputs = self.bert(
2025-09-07T08:07:07.2326868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2327242Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2327612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2327982Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2328314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2328672Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2329058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2329525Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2329946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2330323Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2330675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2331059Z     self_outputs = self.self(
2025-09-07T08:07:07.2331404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2331760Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2332137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2332504Z     self.key(current_states)
2025-09-07T08:07:07.2332611Z 
2025-09-07T08:07:07.2332735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2333167Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2333484Z     return mod(**inputs)
2025-09-07T08:07:07.2333831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2334216Z     outputs = self.bert(
2025-09-07T08:07:07.2334570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2334960Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2335323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2335689Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2336034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2336395Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2336793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2337190Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2337607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2338007Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2338370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2338752Z     self_outputs = self.self(
2025-09-07T08:07:07.2339099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2339470Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2339815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2340190Z     self.value(current_states)
2025-09-07T08:07:07.2340299Z 
2025-09-07T08:07:07.2340377Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2340595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2340932Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2341236Z     return mod(**inputs)
2025-09-07T08:07:07.2341575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2341936Z     outputs = self.bert(
2025-09-07T08:07:07.2342268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2342635Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2342988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2343472Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2343849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2344199Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2344569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2344943Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2345309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2345666Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2346015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2346379Z     self_outputs = self.self(
2025-09-07T08:07:07.2346722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2347144Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2347488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2347911Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2348092Z 
2025-09-07T08:07:07.2348186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2348527Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2348831Z     return mod(**inputs)
2025-09-07T08:07:07.2349168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2349532Z     outputs = self.bert(
2025-09-07T08:07:07.2349862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2350231Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2350586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2350950Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2351277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2351623Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2351985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2352355Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2352722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2353087Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2353444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2353875Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2354289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2354676Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2354815Z 
2025-09-07T08:07:07.2354913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2355261Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2355567Z     return mod(**inputs)
2025-09-07T08:07:07.2355911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2356278Z     outputs = self.bert(
2025-09-07T08:07:07.2356617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2357087Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2357461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2357840Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2358181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2358535Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2358907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2359288Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2359685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2360080Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2360568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2361028Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2361450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2361835Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2361973Z 
2025-09-07T08:07:07.2362074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2362421Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2362731Z     return mod(**inputs)
2025-09-07T08:07:07.2363091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2363459Z     outputs = self.bert(
2025-09-07T08:07:07.2363797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2364174Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2364536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2364906Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2365240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2365589Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2365952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2366329Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2366717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2367104Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2367508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2367952Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2368385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2368802Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2369182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2369517Z     return self.act(input)
2025-09-07T08:07:07.2369629Z 
2025-09-07T08:07:07.2369726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2370080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2370400Z     return mod(**inputs)
2025-09-07T08:07:07.2370806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2371210Z     outputs = self.bert(
2025-09-07T08:07:07.2371565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2371945Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2372316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2372691Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2373024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2373381Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2373758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2374146Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2374602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2375002Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2375407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2375871Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2376300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2376676Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2376812Z 
2025-09-07T08:07:07.2376906Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2377256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2377604Z     return mod(**inputs)
2025-09-07T08:07:07.2377946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2378325Z     outputs = self.bert(
2025-09-07T08:07:07.2378663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2379032Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2379387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2379749Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2380077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2380429Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2380803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2381186Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2381559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2381635Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2381876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2381943Z     self_outputs = self.self(
2025-09-07T08:07:07.2382180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2382240Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2382483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2382690Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2382765Z 
2025-09-07T08:07:07.2382869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2383105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2383174Z     return mod(**inputs)
2025-09-07T08:07:07.2383423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2383492Z     outputs = self.bert(
2025-09-07T08:07:07.2383747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2383821Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2384070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2384131Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2384343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2384417Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2384746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2384820Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2385058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2385118Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2385362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2385426Z     self_outputs = self.self(
2025-09-07T08:07:07.2385663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2385721Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2385967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2386032Z     self.key(current_states)
2025-09-07T08:07:07.2386036Z 
2025-09-07T08:07:07.2386125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2386312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2386368Z     return mod(**inputs)
2025-09-07T08:07:07.2386617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2386676Z     outputs = self.bert(
2025-09-07T08:07:07.2386920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2386990Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2387232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2387306Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2387522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2387601Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2387874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2387944Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2388218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2388284Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2388529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2388590Z     self_outputs = self.self(
2025-09-07T08:07:07.2388825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2388973Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2389215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2389282Z     self.value(current_states)
2025-09-07T08:07:07.2389285Z 
2025-09-07T08:07:07.2389357Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2389447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2389633Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2389691Z     return mod(**inputs)
2025-09-07T08:07:07.2389933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2389992Z     outputs = self.bert(
2025-09-07T08:07:07.2390234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2390303Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2390606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2390690Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2390920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2390990Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2391252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2391321Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2391585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2391662Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2391916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2391995Z     self_outputs = self.self(
2025-09-07T08:07:07.2392231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2392307Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2392548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2392693Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2392696Z 
2025-09-07T08:07:07.2392784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2392983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2393060Z     return mod(**inputs)
2025-09-07T08:07:07.2393308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2393369Z     outputs = self.bert(
2025-09-07T08:07:07.2393612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2393674Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2393915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2393975Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2394204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2394276Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2394518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2394604Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2394902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2394999Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2395241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2395384Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2395624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2395699Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2395705Z 
2025-09-07T08:07:07.2395797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2395991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2396058Z     return mod(**inputs)
2025-09-07T08:07:07.2396313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2396442Z     outputs = self.bert(
2025-09-07T08:07:07.2396693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2396758Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2397013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2397077Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2397294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2397365Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2397606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2397702Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2397971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2398047Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2398326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2398442Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2398689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2398760Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2398764Z 
2025-09-07T08:07:07.2398881Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2399072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2399134Z     return mod(**inputs)
2025-09-07T08:07:07.2399383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2399444Z     outputs = self.bert(
2025-09-07T08:07:07.2399694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2399774Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2400024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2400087Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2400302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2400379Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2400622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2400704Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2401012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2401121Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2401398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2401508Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2401756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2401858Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2402091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2402156Z     return self.act(input)
2025-09-07T08:07:07.2402159Z 
2025-09-07T08:07:07.2402251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2402545Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2402604Z     return mod(**inputs)
2025-09-07T08:07:07.2402852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2402924Z     outputs = self.bert(
2025-09-07T08:07:07.2403314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2403381Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2403621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2403686Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2403910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2403990Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2404236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2404310Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2404574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2404642Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2404922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2405047Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2405288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2405366Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2405372Z 
2025-09-07T08:07:07.2405463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2405656Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2405712Z     return mod(**inputs)
2025-09-07T08:07:07.2405975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2406030Z     outputs = self.bert(
2025-09-07T08:07:07.2406271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2406335Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2406573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2406641Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2406853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2407014Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2407329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2407413Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2407662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2407725Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2407968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2408052Z     self_outputs = self.self(
2025-09-07T08:07:07.2408291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2408353Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2408594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2408916Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2408919Z 
2025-09-07T08:07:07.2409017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2409210Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2409278Z     return mod(**inputs)
2025-09-07T08:07:07.2409528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2409593Z     outputs = self.bert(
2025-09-07T08:07:07.2409839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2409907Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2410159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2410231Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2410457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2410527Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2410775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2410847Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2411088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2411165Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2411406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2411468Z     self_outputs = self.self(
2025-09-07T08:07:07.2411708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2411771Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2412035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2412098Z     self.key(current_states)
2025-09-07T08:07:07.2412101Z 
2025-09-07T08:07:07.2412193Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2412376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2412431Z     return mod(**inputs)
2025-09-07T08:07:07.2412695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2412753Z     outputs = self.bert(
2025-09-07T08:07:07.2412996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2413142Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2413386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2413453Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2413667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2413744Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2413986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2414065Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2414301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2414363Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2414612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2414742Z     self_outputs = self.self(
2025-09-07T08:07:07.2414988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2415048Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2415291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2415363Z     self.value(current_states)
2025-09-07T08:07:07.2415366Z 
2025-09-07T08:07:07.2415439Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2415537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2415718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2415776Z     return mod(**inputs)
2025-09-07T08:07:07.2416031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2416097Z     outputs = self.bert(
2025-09-07T08:07:07.2416349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2416415Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2416661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2416723Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2416937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2417013Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2417252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2417329Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2417619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2417684Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2417934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2417995Z     self_outputs = self.self(
2025-09-07T08:07:07.2418241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2418300Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2418543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2418671Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2418674Z 
2025-09-07T08:07:07.2418764Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2419004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2419100Z     return mod(**inputs)
2025-09-07T08:07:07.2419357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2419416Z     outputs = self.bert(
2025-09-07T08:07:07.2419662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2419735Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2419977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2420046Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2420259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2420327Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2420640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2420714Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2420960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2421025Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2421266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2421394Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2421640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2421724Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2421727Z 
2025-09-07T08:07:07.2421817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2422011Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2422069Z     return mod(**inputs)
2025-09-07T08:07:07.2422312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2422400Z     outputs = self.bert(
2025-09-07T08:07:07.2422644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2422721Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2422964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2423027Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2423247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2423321Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2423571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2423647Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2423916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2423988Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2424265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2424382Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2424628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2424711Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2424714Z 
2025-09-07T08:07:07.2424808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2425040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2425139Z     return mod(**inputs)
2025-09-07T08:07:07.2425387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2425456Z     outputs = self.bert(
2025-09-07T08:07:07.2425703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2425768Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2426019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2426082Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2426304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2426374Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2426681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2426756Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2427015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2427089Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2427362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2427489Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2427735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2427834Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2428054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2428122Z     return self.act(input)
2025-09-07T08:07:07.2428125Z 
2025-09-07T08:07:07.2428224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2428471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2428606Z     return mod(**inputs)
2025-09-07T08:07:07.2428886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2428976Z     outputs = self.bert(
2025-09-07T08:07:07.2429282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2429377Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2429813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2429948Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2430197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2430295Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2430596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2430686Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2431074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2431170Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2431505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2431657Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2432028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2432219Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2432223Z 
2025-09-07T08:07:07.2432355Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2432602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2432688Z     return mod(**inputs)
2025-09-07T08:07:07.2432979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2433083Z     outputs = self.bert(
2025-09-07T08:07:07.2433397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2433534Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2433806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2433944Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2434249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2434335Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2434695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2434796Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2435110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2435203Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2435474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2435630Z     self_outputs = self.self(
2025-09-07T08:07:07.2435917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2436057Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2436352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2436612Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2436616Z 
2025-09-07T08:07:07.2436723Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2436985Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2437127Z     return mod(**inputs)
2025-09-07T08:07:07.2437406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2437522Z     outputs = self.bert(
2025-09-07T08:07:07.2437797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2437970Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2438260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2438367Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2438647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2438745Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2439034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2439176Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2439472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2439594Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2439911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2440064Z     self_outputs = self.self(
2025-09-07T08:07:07.2440323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2440507Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2440787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2440908Z     self.key(current_states)
2025-09-07T08:07:07.2440911Z 
2025-09-07T08:07:07.2441066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2441283Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2441443Z     return mod(**inputs)
2025-09-07T08:07:07.2441731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2441881Z     outputs = self.bert(
2025-09-07T08:07:07.2442200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2442296Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2442599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2442733Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2450623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2450717Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2451004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2451086Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2451371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2451446Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2451701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2451776Z     self_outputs = self.self(
2025-09-07T08:07:07.2452018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2452086Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2452342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2452407Z     self.value(current_states)
2025-09-07T08:07:07.2452412Z 
2025-09-07T08:07:07.2452494Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2452596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2452808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2452874Z     return mod(**inputs)
2025-09-07T08:07:07.2453145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2453211Z     outputs = self.bert(
2025-09-07T08:07:07.2453460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2453536Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2453779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2453842Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2454066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2454144Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2454515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2454633Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2454883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2454947Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2455189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2455258Z     self_outputs = self.self(
2025-09-07T08:07:07.2455494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2455555Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2455792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2455921Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2455927Z 
2025-09-07T08:07:07.2456092Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2456291Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2456352Z     return mod(**inputs)
2025-09-07T08:07:07.2456600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2456660Z     outputs = self.bert(
2025-09-07T08:07:07.2456903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2456971Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2457216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2457280Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2457508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2457643Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2457884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2457957Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2458194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2458261Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2458500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2458619Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2458860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2458937Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2458945Z 
2025-09-07T08:07:07.2459043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2459232Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2459292Z     return mod(**inputs)
2025-09-07T08:07:07.2459535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2459594Z     outputs = self.bert(
2025-09-07T08:07:07.2459843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2459911Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2460156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2460216Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2460488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2460599Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2460841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2460916Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2461179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2461248Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2461542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2461658Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2461908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2462048Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2462053Z 
2025-09-07T08:07:07.2462150Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2462340Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2462396Z     return mod(**inputs)
2025-09-07T08:07:07.2462648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2462707Z     outputs = self.bert(
2025-09-07T08:07:07.2462960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2463025Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2463266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2463337Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2463555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2463635Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2463877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2463956Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2464219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2464287Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2464566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2464673Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2464923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2465032Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2465242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2465312Z     return self.act(input)
2025-09-07T08:07:07.2465316Z 
2025-09-07T08:07:07.2465414Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2465609Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2465667Z     return mod(**inputs)
2025-09-07T08:07:07.2465913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2465975Z     outputs = self.bert(
2025-09-07T08:07:07.2466219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2466353Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2466635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2466705Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2466925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2466999Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2467254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2467331Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2467594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2467662Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2467942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2468153Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2468406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2468490Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2468494Z 
2025-09-07T08:07:07.2468593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2468791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2468849Z     return mod(**inputs)
2025-09-07T08:07:07.2469099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2469163Z     outputs = self.bert(
2025-09-07T08:07:07.2469408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2469481Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2469722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2469785Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2470005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2470074Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2470322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2470393Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2470639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2470704Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2470956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2471024Z     self_outputs = self.self(
2025-09-07T08:07:07.2471263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2471328Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2471570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2471778Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2471781Z 
2025-09-07T08:07:07.2471881Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2472075Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2472134Z     return mod(**inputs)
2025-09-07T08:07:07.2472424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2472513Z     outputs = self.bert(
2025-09-07T08:07:07.2472762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2472830Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2473072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2473140Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2473363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2473436Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2473677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2473755Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2474052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2474122Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2474362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2474424Z     self_outputs = self.self(
2025-09-07T08:07:07.2474668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2474729Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2474974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2475035Z     self.key(current_states)
2025-09-07T08:07:07.2475039Z 
2025-09-07T08:07:07.2475130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2475328Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2475390Z     return mod(**inputs)
2025-09-07T08:07:07.2475638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2475695Z     outputs = self.bert(
2025-09-07T08:07:07.2475945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2476010Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2476249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2476316Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2476527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2476596Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2476840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2476913Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2477153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2477211Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2477455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2477516Z     self_outputs = self.self(
2025-09-07T08:07:07.2477750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2477810Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2478050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2478154Z     self.value(current_states)
2025-09-07T08:07:07.2478186Z 
2025-09-07T08:07:07.2478262Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2478355Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2478540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2478599Z     return mod(**inputs)
2025-09-07T08:07:07.2478851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2478907Z     outputs = self.bert(
2025-09-07T08:07:07.2479154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2479218Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2479458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2479525Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2479796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2479868Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2480111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2480182Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2480421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2480481Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2480720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2480780Z     self_outputs = self.self(
2025-09-07T08:07:07.2481013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2481077Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2481319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2481447Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2481450Z 
2025-09-07T08:07:07.2481539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2481729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2481783Z     return mod(**inputs)
2025-09-07T08:07:07.2482026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2482086Z     outputs = self.bert(
2025-09-07T08:07:07.2482329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2482395Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2482637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2482699Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2482912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2482976Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2483216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2483288Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2483519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2483579Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2483817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2483987Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2484266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2484342Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2484346Z 
2025-09-07T08:07:07.2484434Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2484618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2484676Z     return mod(**inputs)
2025-09-07T08:07:07.2484919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2484976Z     outputs = self.bert(
2025-09-07T08:07:07.2485218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2485281Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2485590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2485652Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2485867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2485933Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2486174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2486247Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2486507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2486577Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2486852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2486972Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2487212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2487283Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2487287Z 
2025-09-07T08:07:07.2487377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2487561Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2487619Z     return mod(**inputs)
2025-09-07T08:07:07.2487860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2487920Z     outputs = self.bert(
2025-09-07T08:07:07.2488166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2488229Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2488474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2488534Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2488748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2488813Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2489052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2489128Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2489385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2489455Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2489731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2489941Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2490184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2490287Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2490499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2490561Z     return self.act(input)
2025-09-07T08:07:07.2490565Z 
2025-09-07T08:07:07.2490656Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2490842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2490899Z     return mod(**inputs)
2025-09-07T08:07:07.2491146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2491205Z     outputs = self.bert(
2025-09-07T08:07:07.2491517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2491581Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2491824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2491892Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2492104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2492175Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2492414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2492487Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2492749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2492815Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2493093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2493218Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2493467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2493541Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2493544Z 
2025-09-07T08:07:07.2493636Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2493825Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2493882Z     return mod(**inputs)
2025-09-07T08:07:07.2494128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2494192Z     outputs = self.bert(
2025-09-07T08:07:07.2494437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2494506Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2494747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2494814Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2495032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2495099Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2495339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2495409Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2495694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2495786Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2496029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2496092Z     self_outputs = self.self(
2025-09-07T08:07:07.2496328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2496388Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2496628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2496829Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2496833Z 
2025-09-07T08:07:07.2496927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2497173Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2497233Z     return mod(**inputs)
2025-09-07T08:07:07.2497478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2497571Z     outputs = self.bert(
2025-09-07T08:07:07.2497816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2497880Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2498121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2498180Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2498397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2498464Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2498707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2498778Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2499011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2499072Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2499309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2499369Z     self_outputs = self.self(
2025-09-07T08:07:07.2499604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2499663Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2499901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2499965Z     self.key(current_states)
2025-09-07T08:07:07.2499971Z 
2025-09-07T08:07:07.2500061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2500245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2500300Z     return mod(**inputs)
2025-09-07T08:07:07.2500550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2500605Z     outputs = self.bert(
2025-09-07T08:07:07.2500849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2500912Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2501150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2501218Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2501479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2501980Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2502222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2502292Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2502526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2502586Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2502825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2502888Z     self_outputs = self.self(
2025-09-07T08:07:07.2503316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2503384Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2503805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2503877Z     self.value(current_states)
2025-09-07T08:07:07.2503881Z 
2025-09-07T08:07:07.2503955Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2504052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2504240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2504295Z     return mod(**inputs)
2025-09-07T08:07:07.2504540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2504596Z     outputs = self.bert(
2025-09-07T08:07:07.2504841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2504907Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2505153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2505220Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2505434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2505502Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2505741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2505811Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2506047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2506106Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2506349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2506411Z     self_outputs = self.self(
2025-09-07T08:07:07.2506651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2506711Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2506952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2507079Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2507082Z 
2025-09-07T08:07:07.2507175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2507364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2507420Z     return mod(**inputs)
2025-09-07T08:07:07.2507662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2507773Z     outputs = self.bert(
2025-09-07T08:07:07.2508081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2508153Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2508400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2508465Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2508685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2508759Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2509004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2509075Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2509323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2509389Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2509688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2509816Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2510060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2510142Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2510146Z 
2025-09-07T08:07:07.2510243Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2510438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2510503Z     return mod(**inputs)
2025-09-07T08:07:07.2510754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2510815Z     outputs = self.bert(
2025-09-07T08:07:07.2511062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2511133Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2511372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2511435Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2511653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2511724Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2511964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2512038Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2512298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2512375Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2512652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2512770Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2513012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2513084Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2513089Z 
2025-09-07T08:07:07.2513184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2513371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2513431Z     return mod(**inputs)
2025-09-07T08:07:07.2513673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2513809Z     outputs = self.bert(
2025-09-07T08:07:07.2514053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2514119Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2514362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2514425Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2514643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2514712Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2514951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2515024Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2515283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2515417Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2515695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2515804Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2516045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2516147Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2516359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2516421Z     return self.act(input)
2025-09-07T08:07:07.2516425Z 
2025-09-07T08:07:07.2516518Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2516705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2516765Z     return mod(**inputs)
2025-09-07T08:07:07.2517010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2517065Z     outputs = self.bert(
2025-09-07T08:07:07.2517314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2517376Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2517616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2517685Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2517898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2517969Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2518212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2518287Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2518543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2518609Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2518886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2519012Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2519259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2519330Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2519333Z 
2025-09-07T08:07:07.2519424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2519653Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2519741Z     return mod(**inputs)
2025-09-07T08:07:07.2519998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2520054Z     outputs = self.bert(
2025-09-07T08:07:07.2520298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2520361Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2520600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2520663Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2520875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2520946Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2521243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2521320Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2521568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2521631Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2521876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2521939Z     self_outputs = self.self(
2025-09-07T08:07:07.2522176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2522237Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2522478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2522686Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2522690Z 
2025-09-07T08:07:07.2522779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2522965Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2523021Z     return mod(**inputs)
2025-09-07T08:07:07.2523270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2523330Z     outputs = self.bert(
2025-09-07T08:07:07.2523573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2523636Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2523874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2523937Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2524156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2524224Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2524465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2524535Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2524772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2524840Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2525077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2525141Z     self_outputs = self.self(
2025-09-07T08:07:07.2525374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2525500Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2525739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2525800Z     self.key(current_states)
2025-09-07T08:07:07.2525803Z 
2025-09-07T08:07:07.2525896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2526080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2526141Z     return mod(**inputs)
2025-09-07T08:07:07.2526391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2526449Z     outputs = self.bert(
2025-09-07T08:07:07.2526696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2526760Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2527070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2527133Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2527349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2527418Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2527663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2527735Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2527969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2528033Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2528272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2528337Z     self_outputs = self.self(
2025-09-07T08:07:07.2528579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2528640Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2528885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2528947Z     self.value(current_states)
2025-09-07T08:07:07.2528950Z 
2025-09-07T08:07:07.2529025Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2529117Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2529299Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2529361Z     return mod(**inputs)
2025-09-07T08:07:07.2529605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2529665Z     outputs = self.bert(
2025-09-07T08:07:07.2529919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2529983Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2530227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2530286Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2530504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2530569Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2530807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2530880Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2531115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2531239Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2531479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2531538Z     self_outputs = self.self(
2025-09-07T08:07:07.2531773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2531831Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2532072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2532194Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2532198Z 
2025-09-07T08:07:07.2532285Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2532467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2532524Z     return mod(**inputs)
2025-09-07T08:07:07.2532843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2532900Z     outputs = self.bert(
2025-09-07T08:07:07.2533145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2533209Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2533448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2533515Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2533726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2533794Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2534033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2534110Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2534347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2534405Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2534647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2534765Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2535005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2535081Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2535084Z 
2025-09-07T08:07:07.2535171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2535356Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2535413Z     return mod(**inputs)
2025-09-07T08:07:07.2535663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2535721Z     outputs = self.bert(
2025-09-07T08:07:07.2535962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2536026Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2536267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2536334Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2536547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2536614Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2536861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2537002Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2537288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2537353Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2537697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2537806Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2538046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2538117Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2538121Z 
2025-09-07T08:07:07.2538209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2538397Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2538544Z     return mod(**inputs)
2025-09-07T08:07:07.2538794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2538856Z     outputs = self.bert(
2025-09-07T08:07:07.2539099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2539163Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2539404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2539464Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2539676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2539742Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2539989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2540061Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2540323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2540389Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2540660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2540768Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2541008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2541113Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2541318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2541381Z     return self.act(input)
2025-09-07T08:07:07.2541389Z 
2025-09-07T08:07:07.2541479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2541663Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2541722Z     return mod(**inputs)
2025-09-07T08:07:07.2541965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2542022Z     outputs = self.bert(
2025-09-07T08:07:07.2542264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2542327Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2542569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2542631Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2542901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2543002Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2543242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2543315Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2543569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2543636Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2543906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2544025Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2544269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2544411Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2544415Z 
2025-09-07T08:07:07.2544514Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2544696Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2544760Z     return mod(**inputs)
2025-09-07T08:07:07.2545005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2545061Z     outputs = self.bert(
2025-09-07T08:07:07.2545308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2545372Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2545617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2545678Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2545894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2545965Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2546206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2546282Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2546517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2546585Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2546826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2546889Z     self_outputs = self.self(
2025-09-07T08:07:07.2547132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2547195Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2547443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2547642Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2547645Z 
2025-09-07T08:07:07.2547732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2547917Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2547972Z     return mod(**inputs)
2025-09-07T08:07:07.2548223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2548278Z     outputs = self.bert(
2025-09-07T08:07:07.2548521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2548632Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2548908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2548971Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2549181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2549250Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2549488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2549557Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2549794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2549851Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2550092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2550230Z     self_outputs = self.self(
2025-09-07T08:07:07.2550470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2550531Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2550771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2550836Z     self.key(current_states)
2025-09-07T08:07:07.2550840Z 
2025-09-07T08:07:07.2550930Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2551127Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2551186Z     return mod(**inputs)
2025-09-07T08:07:07.2551431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2551498Z     outputs = self.bert(
2025-09-07T08:07:07.2551745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2551817Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2552061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2552125Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2552340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2552407Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2552652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2552721Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2552952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2553026Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2553266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2553333Z     self_outputs = self.self(
2025-09-07T08:07:07.2553564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2553626Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2553872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2553937Z     self.value(current_states)
2025-09-07T08:07:07.2553939Z 
2025-09-07T08:07:07.2554017Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2554107Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2554297Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2554432Z     return mod(**inputs)
2025-09-07T08:07:07.2554679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2554744Z     outputs = self.bert(
2025-09-07T08:07:07.2554989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2555059Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2555296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2555358Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2555576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2555644Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2555887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2556040Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2556280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2556351Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2556591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2556658Z     self_outputs = self.self(
2025-09-07T08:07:07.2556893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2556960Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2557202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2557321Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2557327Z 
2025-09-07T08:07:07.2557431Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2557619Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2557682Z     return mod(**inputs)
2025-09-07T08:07:07.2557928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2557984Z     outputs = self.bert(
2025-09-07T08:07:07.2558238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2558300Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2558548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2558610Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2558821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2558899Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2559141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2559217Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2559451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2559515Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2559754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2559873Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2560119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2560237Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2560272Z 
2025-09-07T08:07:07.2560371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2560556Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2560614Z     return mod(**inputs)
2025-09-07T08:07:07.2560865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2560922Z     outputs = self.bert(
2025-09-07T08:07:07.2561168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2561232Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2561474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2561542Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2561751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2561894Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2562143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2562220Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2562478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2562550Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2562835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2562943Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2563189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2563263Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2563268Z 
2025-09-07T08:07:07.2563365Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2563552Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2563608Z     return mod(**inputs)
2025-09-07T08:07:07.2563854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2563911Z     outputs = self.bert(
2025-09-07T08:07:07.2564160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2564225Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2564466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2564531Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2564746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2564822Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2565061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2565132Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2565394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2565460Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2565738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2565843Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2566093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2566260Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2566469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2566537Z     return self.act(input)
2025-09-07T08:07:07.2566540Z 
2025-09-07T08:07:07.2566633Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2566824Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2566882Z     return mod(**inputs)
2025-09-07T08:07:07.2567138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2567208Z     outputs = self.bert(
2025-09-07T08:07:07.2567455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2567528Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2567834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2567905Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2568133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2568208Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2568457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2568536Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2568801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2568874Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2569152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2569291Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2569537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2569616Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2569620Z 
2025-09-07T08:07:07.2569718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2569911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2569977Z     return mod(**inputs)
2025-09-07T08:07:07.2570227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2570290Z     outputs = self.bert(
2025-09-07T08:07:07.2570534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2570608Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2570854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2570920Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2571141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2571214Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2571462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2571532Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2571776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2571864Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2572107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2572245Z     self_outputs = self.self(
2025-09-07T08:07:07.2572484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2572547Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2572798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2573000Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2573003Z 
2025-09-07T08:07:07.2573104Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2573293Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2573358Z     return mod(**inputs)
2025-09-07T08:07:07.2573605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2573667Z     outputs = self.bert(
2025-09-07T08:07:07.2573975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2574040Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2574289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2574352Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2574572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2574647Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2574884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2574961Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2575201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2575266Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2575513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2575575Z     self_outputs = self.self(
2025-09-07T08:07:07.2575815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2575877Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2576122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2576185Z     self.key(current_states)
2025-09-07T08:07:07.2576189Z 
2025-09-07T08:07:07.2576279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2576473Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2576535Z     return mod(**inputs)
2025-09-07T08:07:07.2576792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2576850Z     outputs = self.bert(
2025-09-07T08:07:07.2577092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2577156Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2577394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2577456Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2577720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2577785Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2578028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2578191Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2578429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2578487Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2578728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2578788Z     self_outputs = self.self(
2025-09-07T08:07:07.2579020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2579081Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2579318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2579384Z     self.value(current_states)
2025-09-07T08:07:07.2579389Z 
2025-09-07T08:07:07.2579461Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2579611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2579800Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2579856Z     return mod(**inputs)
2025-09-07T08:07:07.2580103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2580158Z     outputs = self.bert(
2025-09-07T08:07:07.2580400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2580463Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2580702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2580764Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2580974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2581048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2581285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2581354Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2581588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2581645Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2581886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2581945Z     self_outputs = self.self(
2025-09-07T08:07:07.2582177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2582237Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2582477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2582603Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2582606Z 
2025-09-07T08:07:07.2582693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2582876Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2582932Z     return mod(**inputs)
2025-09-07T08:07:07.2583174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2583231Z     outputs = self.bert(
2025-09-07T08:07:07.2583474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2583537Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2583775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2583904Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2584120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2584184Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2584425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2584494Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2584729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2584790Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2585028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2585149Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2585447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2585521Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2585527Z 
2025-09-07T08:07:07.2585614Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2585797Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2585855Z     return mod(**inputs)
2025-09-07T08:07:07.2586098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2586155Z     outputs = self.bert(
2025-09-07T08:07:07.2586395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2586457Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2586701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2586763Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2586976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2587042Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2587281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2587356Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2587615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2587683Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2587960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2588077Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2588319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2588389Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2588393Z 
2025-09-07T08:07:07.2588482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2588665Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2588721Z     return mod(**inputs)
2025-09-07T08:07:07.2588961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2589017Z     outputs = self.bert(
2025-09-07T08:07:07.2589259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2589320Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2589618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2589711Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2589923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2589991Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2590229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2590302Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2590556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2590623Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2590895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2591083Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2591333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2591439Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2591648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2591711Z     return self.act(input)
2025-09-07T08:07:07.2591714Z 
2025-09-07T08:07:07.2591803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2591987Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2592042Z     return mod(**inputs)
2025-09-07T08:07:07.2592287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2592345Z     outputs = self.bert(
2025-09-07T08:07:07.2592591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2592654Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2592892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2592955Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2593172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2593244Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2593482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2593552Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2593809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2593877Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2594151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2594273Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2594516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2594588Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2594590Z 
2025-09-07T08:07:07.2594678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2594862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2594917Z     return mod(**inputs)
2025-09-07T08:07:07.2595161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2595322Z     outputs = self.bert(
2025-09-07T08:07:07.2595570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2595634Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2595874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2595941Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2596151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2596215Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2596455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2596527Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2596767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2596889Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2597132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2597196Z     self_outputs = self.self(
2025-09-07T08:07:07.2597431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2597492Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2597730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2597928Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2597932Z 
2025-09-07T08:07:07.2598019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2598202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2598263Z     return mod(**inputs)
2025-09-07T08:07:07.2598506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2598564Z     outputs = self.bert(
2025-09-07T08:07:07.2598806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2598867Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2599107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2599168Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2599380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2599444Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2599686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2599756Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2599989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2600050Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2600288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2600349Z     self_outputs = self.self(
2025-09-07T08:07:07.2600582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2600643Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2600889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2601011Z     self.key(current_states)
2025-09-07T08:07:07.2601014Z 
2025-09-07T08:07:07.2601143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2601329Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2601391Z     return mod(**inputs)
2025-09-07T08:07:07.2601643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2601700Z     outputs = self.bert(
2025-09-07T08:07:07.2601952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2602014Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2602265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2602327Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2602539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2602686Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2602931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2603008Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2603429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2603493Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2603751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2603814Z     self_outputs = self.self(
2025-09-07T08:07:07.2604062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2604122Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2604375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2604447Z     self.value(current_states)
2025-09-07T08:07:07.2604451Z 
2025-09-07T08:07:07.2604523Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2604624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2604820Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2604884Z     return mod(**inputs)
2025-09-07T08:07:07.2605135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2605194Z     outputs = self.bert(
2025-09-07T08:07:07.2605447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2605509Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2605759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2605825Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2606044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2606121Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2606361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2606437Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2606675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2606743Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2606990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2607145Z     self_outputs = self.self(
2025-09-07T08:07:07.2607451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2607513Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2607759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2607884Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2607887Z 
2025-09-07T08:07:07.2607979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2608169Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2608224Z     return mod(**inputs)
2025-09-07T08:07:07.2608474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2608529Z     outputs = self.bert(
2025-09-07T08:07:07.2608866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2608935Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2609178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2609242Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2609457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2609525Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2609767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2609836Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2610074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2610136Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2610381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2610499Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2610737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2610811Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2610814Z 
2025-09-07T08:07:07.2610903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2611092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2611146Z     return mod(**inputs)
2025-09-07T08:07:07.2611389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2611447Z     outputs = self.bert(
2025-09-07T08:07:07.2611699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2611768Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2612009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2612074Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2612291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2612358Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2612601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2612674Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2612941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2613047Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2613364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2613478Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2613719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2613792Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2613796Z 
2025-09-07T08:07:07.2613886Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2614072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2614130Z     return mod(**inputs)
2025-09-07T08:07:07.2614373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2614433Z     outputs = self.bert(
2025-09-07T08:07:07.2614736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2614803Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2615045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2615105Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2615321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2615387Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2615629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2615698Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2615953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2616025Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2616298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2616404Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2616642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2616742Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2616949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2617009Z     return self.act(input)
2025-09-07T08:07:07.2617012Z 
2025-09-07T08:07:07.2617104Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2617291Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2617350Z     return mod(**inputs)
2025-09-07T08:07:07.2617642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2617699Z     outputs = self.bert(
2025-09-07T08:07:07.2617945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2618010Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2618253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2618312Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2618527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2618596Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2618837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2618991Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2619250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2619321Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2619595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2619715Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2619962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2620033Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2620036Z 
2025-09-07T08:07:07.2620134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2620317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2620457Z     return mod(**inputs)
2025-09-07T08:07:07.2620716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2620775Z     outputs = self.bert(
2025-09-07T08:07:07.2621022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2621086Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2621329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2621399Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2621611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2621686Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2621930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2622013Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2622252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2622316Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2622560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2622624Z     self_outputs = self.self(
2025-09-07T08:07:07.2622863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2622922Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2623163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2623363Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2623373Z 
2025-09-07T08:07:07.2623462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2623649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2623705Z     return mod(**inputs)
2025-09-07T08:07:07.2623953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2624009Z     outputs = self.bert(
2025-09-07T08:07:07.2624252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2624320Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2624559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2624623Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2624880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2624981Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2625226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2625296Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2625533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2625591Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2625830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2625893Z     self_outputs = self.self(
2025-09-07T08:07:07.2626127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2626192Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2626490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2626557Z     self.key(current_states)
2025-09-07T08:07:07.2626560Z 
2025-09-07T08:07:07.2626648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2626833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2626894Z     return mod(**inputs)
2025-09-07T08:07:07.2627137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2627195Z     outputs = self.bert(
2025-09-07T08:07:07.2627437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2627500Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2627744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2627811Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2628027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2628091Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2628333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2628407Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2628644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2628705Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2628943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2629008Z     self_outputs = self.self(
2025-09-07T08:07:07.2629244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2629304Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2629545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2629606Z     self.value(current_states)
2025-09-07T08:07:07.2629609Z 
2025-09-07T08:07:07.2629683Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2629768Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2629948Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2630004Z     return mod(**inputs)
2025-09-07T08:07:07.2630249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2630310Z     outputs = self.bert(
2025-09-07T08:07:07.2630596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2630694Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2630936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2630996Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2631213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2631277Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2631517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2631587Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2631820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2631883Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2632183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2632248Z     self_outputs = self.self(
2025-09-07T08:07:07.2632479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2632538Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2632779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2632900Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2632904Z 
2025-09-07T08:07:07.2632994Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2633174Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2633230Z     return mod(**inputs)
2025-09-07T08:07:07.2633480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2633537Z     outputs = self.bert(
2025-09-07T08:07:07.2633784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2633843Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2634084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2634146Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2634356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2634423Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2634661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2634735Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2634972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2635032Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2635271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2635386Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2635630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2635703Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2635705Z 
2025-09-07T08:07:07.2635793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2635979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2636069Z     return mod(**inputs)
2025-09-07T08:07:07.2636319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2636409Z     outputs = self.bert(
2025-09-07T08:07:07.2636657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2636719Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2636961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2637024Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2637236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2637303Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2637541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2637614Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2637938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2638004Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2638284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2638393Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2638634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2638708Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2638711Z 
2025-09-07T08:07:07.2638798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2638984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2639040Z     return mod(**inputs)
2025-09-07T08:07:07.2639289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2639343Z     outputs = self.bert(
2025-09-07T08:07:07.2639585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2639648Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2639889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2639950Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2640159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2640224Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2640469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2640545Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2640801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2640864Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2641136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2641238Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2641477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2641577Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2641781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2641882Z     return self.act(input)
2025-09-07T08:07:07.2641885Z 
2025-09-07T08:07:07.2642006Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2642190Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2642250Z     return mod(**inputs)
2025-09-07T08:07:07.2642496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2642554Z     outputs = self.bert(
2025-09-07T08:07:07.2642794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2642855Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2643101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2643160Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2643373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2643516Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2643765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2643834Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2644086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2644152Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2644424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2644549Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2644792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2644862Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2644870Z 
2025-09-07T08:07:07.2644962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2645142Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2645199Z     return mod(**inputs)
2025-09-07T08:07:07.2645448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2645506Z     outputs = self.bert(
2025-09-07T08:07:07.2645750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2645811Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2646054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2646115Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2646331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2646398Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2646638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2646712Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2646949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2647010Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2647250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2647309Z     self_outputs = self.self(
2025-09-07T08:07:07.2647546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2647641Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2647932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2648128Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2648132Z 
2025-09-07T08:07:07.2648225Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2648408Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2648467Z     return mod(**inputs)
2025-09-07T08:07:07.2648713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2648769Z     outputs = self.bert(
2025-09-07T08:07:07.2649014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2649075Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2649525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2649590Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2649802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2649870Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2650110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2650183Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2650417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2650474Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2650719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2650793Z     self_outputs = self.self(
2025-09-07T08:07:07.2651035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2651092Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2651330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2651393Z     self.key(current_states)
2025-09-07T08:07:07.2651397Z 
2025-09-07T08:07:07.2651490Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2651676Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2651730Z     return mod(**inputs)
2025-09-07T08:07:07.2651973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2652031Z     outputs = self.bert(
2025-09-07T08:07:07.2652278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2652344Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2652583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2652644Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2652859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2652923Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2653162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2653230Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2653469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2653570Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2653846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2653909Z     self_outputs = self.self(
2025-09-07T08:07:07.2654142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2654201Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2654436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2654498Z     self.value(current_states)
2025-09-07T08:07:07.2654501Z 
2025-09-07T08:07:07.2654574Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2654660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2654844Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2654899Z     return mod(**inputs)
2025-09-07T08:07:07.2655202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2655267Z     outputs = self.bert(
2025-09-07T08:07:07.2655512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2655577Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2655818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2655880Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2656089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2656152Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2656395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2656468Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2656710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2656766Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2657007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2657069Z     self_outputs = self.self(
2025-09-07T08:07:07.2657303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2657361Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2657651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2657770Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2657776Z 
2025-09-07T08:07:07.2657864Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2658052Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2658113Z     return mod(**inputs)
2025-09-07T08:07:07.2658357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2658414Z     outputs = self.bert(
2025-09-07T08:07:07.2658660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2658721Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2658965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2659023Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2659236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2659340Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2659620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2659694Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2659930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2659992Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2660228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2660343Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2660589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2660661Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2660667Z 
2025-09-07T08:07:07.2660759Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2661004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2661069Z     return mod(**inputs)
2025-09-07T08:07:07.2661312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2661368Z     outputs = self.bert(
2025-09-07T08:07:07.2661614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2661675Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2661917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2661979Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2662197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2662270Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2662512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2662586Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2662843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2662911Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2663188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2663294Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2663534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2663601Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2663606Z 
2025-09-07T08:07:07.2663699Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2663880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2663935Z     return mod(**inputs)
2025-09-07T08:07:07.2664178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2664232Z     outputs = self.bert(
2025-09-07T08:07:07.2664475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2664535Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2664772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2664834Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2665042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2665197Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2665437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2665507Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2665761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2665825Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2666103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2666205Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2666445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2666547Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2666815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2666881Z     return self.act(input)
2025-09-07T08:07:07.2666884Z 
2025-09-07T08:07:07.2666973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2667167Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2667223Z     return mod(**inputs)
2025-09-07T08:07:07.2667475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2667532Z     outputs = self.bert(
2025-09-07T08:07:07.2667775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2667842Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2668083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2668156Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2668371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2668438Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2668687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2668759Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2669022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2669088Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2669362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2669493Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2669752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2669829Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2669833Z 
2025-09-07T08:07:07.2669923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2670115Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2670173Z     return mod(**inputs)
2025-09-07T08:07:07.2670423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2670487Z     outputs = self.bert(
2025-09-07T08:07:07.2670734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2670807Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2671087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2671180Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2671399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2671468Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2671720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2671791Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2672036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2672100Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2672341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2672412Z     self_outputs = self.self(
2025-09-07T08:07:07.2672711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2672777Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2673020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2673224Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2673227Z 
2025-09-07T08:07:07.2673321Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2673505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2673569Z     return mod(**inputs)
2025-09-07T08:07:07.2673906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2673994Z     outputs = self.bert(
2025-09-07T08:07:07.2674301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2680883Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2681210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2681280Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2681530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2681606Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2681866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2681942Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2682190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2682265Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2682518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2682588Z     self_outputs = self.self(
2025-09-07T08:07:07.2682824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2682885Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2683131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2683193Z     self.key(current_states)
2025-09-07T08:07:07.2683197Z 
2025-09-07T08:07:07.2683300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2683500Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2683634Z     return mod(**inputs)
2025-09-07T08:07:07.2683894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2683991Z     outputs = self.bert(
2025-09-07T08:07:07.2684251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2684320Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2684581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2684647Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2684865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2684944Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2685186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2685265Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2685577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2685638Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2685881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2685943Z     self_outputs = self.self(
2025-09-07T08:07:07.2686182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2686240Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2686485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2686549Z     self.value(current_states)
2025-09-07T08:07:07.2686552Z 
2025-09-07T08:07:07.2686624Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2686726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2686925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2686984Z     return mod(**inputs)
2025-09-07T08:07:07.2687233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2687291Z     outputs = self.bert(
2025-09-07T08:07:07.2687547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2687609Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2687856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2687919Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2688132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2688202Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2688442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2688515Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2688750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2688811Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2689054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2689112Z     self_outputs = self.self(
2025-09-07T08:07:07.2689350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2689408Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2689653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2689847Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2689851Z 
2025-09-07T08:07:07.2689943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2690143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2690199Z     return mod(**inputs)
2025-09-07T08:07:07.2690450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2690506Z     outputs = self.bert(
2025-09-07T08:07:07.2690750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2690817Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2691060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2691133Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2691407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2691479Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2691724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2691793Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2692031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2692090Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2692334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2692454Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2692698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2692776Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2692780Z 
2025-09-07T08:07:07.2692869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2693056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2693110Z     return mod(**inputs)
2025-09-07T08:07:07.2693363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2693421Z     outputs = self.bert(
2025-09-07T08:07:07.2693678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2693743Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2693984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2694052Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2694270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2694336Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2694576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2694647Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2694911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2694979Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2695256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2695370Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2695653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2695758Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2695761Z 
2025-09-07T08:07:07.2695854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2696047Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2696102Z     return mod(**inputs)
2025-09-07T08:07:07.2696349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2696408Z     outputs = self.bert(
2025-09-07T08:07:07.2696648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2696711Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2696953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2697076Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2697296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2697365Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2697688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2697762Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2698018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2698082Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2698353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2698470Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2698714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2698820Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2699030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2699089Z     return self.act(input)
2025-09-07T08:07:07.2699092Z 
2025-09-07T08:07:07.2699185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2699371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2699429Z     return mod(**inputs)
2025-09-07T08:07:07.2699673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2699729Z     outputs = self.bert(
2025-09-07T08:07:07.2699986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2700049Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2700292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2700353Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2700567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2700634Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2700873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2700944Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2701195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2701305Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2701627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2701754Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2702003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2702072Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2702075Z 
2025-09-07T08:07:07.2702172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2702355Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2702411Z     return mod(**inputs)
2025-09-07T08:07:07.2702658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2702716Z     outputs = self.bert(
2025-09-07T08:07:07.2703021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2703250Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2703495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2703555Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2703766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2703832Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2704071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2704143Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2704383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2704445Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2704695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2704754Z     self_outputs = self.self(
2025-09-07T08:07:07.2704990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2705048Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2705288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2705495Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2705499Z 
2025-09-07T08:07:07.2705587Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2705771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2705826Z     return mod(**inputs)
2025-09-07T08:07:07.2706076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2706131Z     outputs = self.bert(
2025-09-07T08:07:07.2706377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2706446Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2706686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2706754Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2706966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2707031Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2707278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2707467Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2707713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2707773Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2708020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2708080Z     self_outputs = self.self(
2025-09-07T08:07:07.2708315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2708381Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2708622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2708689Z     self.key(current_states)
2025-09-07T08:07:07.2708693Z 
2025-09-07T08:07:07.2708784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2709068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2709134Z     return mod(**inputs)
2025-09-07T08:07:07.2709393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2709458Z     outputs = self.bert(
2025-09-07T08:07:07.2709704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2709767Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2710013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2710075Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2710295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2710363Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2710606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2710675Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2710907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2710965Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2711201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2711261Z     self_outputs = self.self(
2025-09-07T08:07:07.2711492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2711550Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2711788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2711853Z     self.value(current_states)
2025-09-07T08:07:07.2711857Z 
2025-09-07T08:07:07.2711929Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2712019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2712201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2712258Z     return mod(**inputs)
2025-09-07T08:07:07.2712500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2712556Z     outputs = self.bert(
2025-09-07T08:07:07.2712797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2712860Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2713100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2713226Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2713445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2713510Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2713751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2713818Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2714053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2714115Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2714356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2714415Z     self_outputs = self.self(
2025-09-07T08:07:07.2714650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2714772Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2715019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2715142Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2715145Z 
2025-09-07T08:07:07.2715240Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2715422Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2715477Z     return mod(**inputs)
2025-09-07T08:07:07.2715722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2715779Z     outputs = self.bert(
2025-09-07T08:07:07.2716025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2716091Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2716329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2716390Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2716602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2716671Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2716910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2716982Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2717212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2717270Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2717517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2717634Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2717885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2717956Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2717959Z 
2025-09-07T08:07:07.2718050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2718240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2718297Z     return mod(**inputs)
2025-09-07T08:07:07.2718555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2718611Z     outputs = self.bert(
2025-09-07T08:07:07.2718859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2718988Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2719236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2719303Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2719519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2719599Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2719843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2719919Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2720188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2720255Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2720603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2720723Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2720968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2721043Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2721047Z 
2025-09-07T08:07:07.2721140Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2721333Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2721390Z     return mod(**inputs)
2025-09-07T08:07:07.2721641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2721698Z     outputs = self.bert(
2025-09-07T08:07:07.2721948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2722020Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2722266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2722338Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2722553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2722625Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2722874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2722945Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2723210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2723279Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2723559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2723665Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2723903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2724009Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2724219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2724289Z     return self.act(input)
2025-09-07T08:07:07.2724292Z 
2025-09-07T08:07:07.2724381Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2724567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2724628Z     return mod(**inputs)
2025-09-07T08:07:07.2724931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2725038Z     outputs = self.bert(
2025-09-07T08:07:07.2725291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2725355Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2725603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2725666Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2725886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2725953Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2726193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2726275Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2726593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2726666Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2726936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2727061Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2727312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2727385Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2727388Z 
2025-09-07T08:07:07.2727485Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2727672Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2727737Z     return mod(**inputs)
2025-09-07T08:07:07.2727989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2728044Z     outputs = self.bert(
2025-09-07T08:07:07.2728290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2728353Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2728599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2728660Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2728876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2728950Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2729189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2729266Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2729508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2729570Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2729815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2729876Z     self_outputs = self.self(
2025-09-07T08:07:07.2730113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2730172Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2730420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2730625Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2730664Z 
2025-09-07T08:07:07.2730785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2730973Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2731029Z     return mod(**inputs)
2025-09-07T08:07:07.2731276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2731330Z     outputs = self.bert(
2025-09-07T08:07:07.2731573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2731641Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2731881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2731948Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2732158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2732490Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2732739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2732813Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2733055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2733114Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2733357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2733416Z     self_outputs = self.self(
2025-09-07T08:07:07.2733652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2733716Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2733959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2734023Z     self.key(current_states)
2025-09-07T08:07:07.2734027Z 
2025-09-07T08:07:07.2734115Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2734301Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2734364Z     return mod(**inputs)
2025-09-07T08:07:07.2734616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2734674Z     outputs = self.bert(
2025-09-07T08:07:07.2734916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2734983Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2735231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2735293Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2735515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2735579Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2735819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2735890Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2736122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2736186Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2736423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2736485Z     self_outputs = self.self(
2025-09-07T08:07:07.2736759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2736853Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2737099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2737160Z     self.value(current_states)
2025-09-07T08:07:07.2737163Z 
2025-09-07T08:07:07.2737239Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2737325Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2737506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2737612Z     return mod(**inputs)
2025-09-07T08:07:07.2737858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2737915Z     outputs = self.bert(
2025-09-07T08:07:07.2738156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2738286Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2738533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2738598Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2738815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2738883Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2739126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2739194Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2739427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2739490Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2739730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2739795Z     self_outputs = self.self(
2025-09-07T08:07:07.2740028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2740087Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2740327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2740448Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2740451Z 
2025-09-07T08:07:07.2740546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2740729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2740788Z     return mod(**inputs)
2025-09-07T08:07:07.2741032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2741092Z     outputs = self.bert(
2025-09-07T08:07:07.2741337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2741399Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2741645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2741705Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2741917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2741986Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2742226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2742302Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2742579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2742669Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2742914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2743034Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2743277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2743349Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2743353Z 
2025-09-07T08:07:07.2743442Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2743626Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2743681Z     return mod(**inputs)
2025-09-07T08:07:07.2743928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2744060Z     outputs = self.bert(
2025-09-07T08:07:07.2744314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2744375Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2744615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2744679Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2744890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2744957Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2745194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2745269Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2745533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2745602Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2745884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2745995Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2746238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2746308Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2746311Z 
2025-09-07T08:07:07.2746400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2746588Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2746646Z     return mod(**inputs)
2025-09-07T08:07:07.2746895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2746953Z     outputs = self.bert(
2025-09-07T08:07:07.2747198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2747265Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2747504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2747567Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2747778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2747846Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2748086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2748192Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2748495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2748564Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2748847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2748949Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2749188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2749293Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2749500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2749567Z     return self.act(input)
2025-09-07T08:07:07.2749570Z 
2025-09-07T08:07:07.2749661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2749910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2749966Z     return mod(**inputs)
2025-09-07T08:07:07.2750215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2750281Z     outputs = self.bert(
2025-09-07T08:07:07.2750523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2750590Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2750828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2750889Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2751106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2751172Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2751417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2751485Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2751743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2751810Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2752081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2752210Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2752452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2752521Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2752526Z 
2025-09-07T08:07:07.2752613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2752799Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2752860Z     return mod(**inputs)
2025-09-07T08:07:07.2753106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2753168Z     outputs = self.bert(
2025-09-07T08:07:07.2753412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2753472Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2753713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2753774Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2753990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2754145Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2754406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2754481Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2754727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2754793Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2755036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2755103Z     self_outputs = self.self(
2025-09-07T08:07:07.2755339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2755401Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2755713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2755921Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2755925Z 
2025-09-07T08:07:07.2756023Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2756212Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2756270Z     return mod(**inputs)
2025-09-07T08:07:07.2756522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2756579Z     outputs = self.bert(
2025-09-07T08:07:07.2756831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2756897Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2757143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2757211Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2757427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2757501Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2757740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2757816Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2758051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2758110Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2758354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2758416Z     self_outputs = self.self(
2025-09-07T08:07:07.2758656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2758716Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2758957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2759023Z     self.key(current_states)
2025-09-07T08:07:07.2759026Z 
2025-09-07T08:07:07.2759119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2759308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2759366Z     return mod(**inputs)
2025-09-07T08:07:07.2759616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2759673Z     outputs = self.bert(
2025-09-07T08:07:07.2759915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2760049Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2760291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2760358Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2760570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2760640Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2760888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2760959Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2761198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2761257Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2761566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2761636Z     self_outputs = self.self(
2025-09-07T08:07:07.2761882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2761947Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2762185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2762252Z     self.value(current_states)
2025-09-07T08:07:07.2762255Z 
2025-09-07T08:07:07.2762329Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2762420Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2762612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2762668Z     return mod(**inputs)
2025-09-07T08:07:07.2762920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2762978Z     outputs = self.bert(
2025-09-07T08:07:07.2763221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2763287Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2763525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2763591Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2763802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2763867Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2764110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2764178Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2764421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2764479Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2764726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2764786Z     self_outputs = self.self(
2025-09-07T08:07:07.2765022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2765084Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2765321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2765447Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2765450Z 
2025-09-07T08:07:07.2765540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2765764Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2765855Z     return mod(**inputs)
2025-09-07T08:07:07.2766100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2766158Z     outputs = self.bert(
2025-09-07T08:07:07.2766400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2766463Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2766702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2766763Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2766981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2767046Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2767362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2767434Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2767668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2767729Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2767967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2768086Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2768323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2768395Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2768398Z 
2025-09-07T08:07:07.2768490Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2768675Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2768735Z     return mod(**inputs)
2025-09-07T08:07:07.2768979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2769038Z     outputs = self.bert(
2025-09-07T08:07:07.2769284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2769347Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2769591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2769650Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2769867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2769934Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2770179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2770257Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2770517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2770589Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2770869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2770982Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2771228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2771299Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2771302Z 
2025-09-07T08:07:07.2771436Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2771690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2771756Z     return mod(**inputs)
2025-09-07T08:07:07.2772005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2772062Z     outputs = self.bert(
2025-09-07T08:07:07.2772309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2772373Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2772619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2772681Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2772895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2772968Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2773270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2773348Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2773605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2773672Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2773954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2774059Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2774305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2774409Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2774625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2774691Z     return self.act(input)
2025-09-07T08:07:07.2774694Z 
2025-09-07T08:07:07.2774782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2774975Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2775033Z     return mod(**inputs)
2025-09-07T08:07:07.2775277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2775334Z     outputs = self.bert(
2025-09-07T08:07:07.2775578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2775645Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2775884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2775954Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2776169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2776239Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2776479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2776549Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2776809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2776874Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2777147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2777274Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2777605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2777710Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2777713Z 
2025-09-07T08:07:07.2777803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2777991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2778047Z     return mod(**inputs)
2025-09-07T08:07:07.2778296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2778354Z     outputs = self.bert(
2025-09-07T08:07:07.2778596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2778663Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2778905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2779033Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2779249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2779317Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2779572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2779644Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2779884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2779944Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2780182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2780246Z     self_outputs = self.self(
2025-09-07T08:07:07.2780484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2780551Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2780790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2780998Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2781002Z 
2025-09-07T08:07:07.2781093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2781276Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2781339Z     return mod(**inputs)
2025-09-07T08:07:07.2781582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2781645Z     outputs = self.bert(
2025-09-07T08:07:07.2781890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2781956Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2782205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2782267Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2782485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2782553Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2782795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2782870Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2783107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2783210Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2783491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2783555Z     self_outputs = self.self(
2025-09-07T08:07:07.2783794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2783855Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2784101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2784162Z     self.key(current_states)
2025-09-07T08:07:07.2784165Z 
2025-09-07T08:07:07.2784259Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2784442Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2784500Z     return mod(**inputs)
2025-09-07T08:07:07.2784750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2784868Z     outputs = self.bert(
2025-09-07T08:07:07.2785120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2785185Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2785428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2785494Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2785707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2785781Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2786021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2786096Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2786338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2786401Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2786648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2786710Z     self_outputs = self.self(
2025-09-07T08:07:07.2786947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2787007Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2787252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2787321Z     self.value(current_states)
2025-09-07T08:07:07.2787325Z 
2025-09-07T08:07:07.2787397Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2787494Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2787679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2787739Z     return mod(**inputs)
2025-09-07T08:07:07.2787989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2788048Z     outputs = self.bert(
2025-09-07T08:07:07.2788295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2788358Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2788603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2788665Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2788874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2788984Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2789277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2789356Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2789593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2789654Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2789902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2789963Z     self_outputs = self.self(
2025-09-07T08:07:07.2790205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2790265Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2790504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2790690Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2790696Z 
2025-09-07T08:07:07.2790785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2790975Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2791031Z     return mod(**inputs)
2025-09-07T08:07:07.2791284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2791342Z     outputs = self.bert(
2025-09-07T08:07:07.2791585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2791652Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2791890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2791959Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2792176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2792241Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2792487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2792556Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2792796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2792853Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2793096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2793219Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2793463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2793552Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2793555Z 
2025-09-07T08:07:07.2793641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2793828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2793885Z     return mod(**inputs)
2025-09-07T08:07:07.2794128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2794191Z     outputs = self.bert(
2025-09-07T08:07:07.2794435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2794501Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2794741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2794841Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2795095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2795163Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2795412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2795487Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2795746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2795816Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2796095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2796211Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2796450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2796589Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2796593Z 
2025-09-07T08:07:07.2796686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2796885Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2796950Z     return mod(**inputs)
2025-09-07T08:07:07.2797208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2797273Z     outputs = self.bert(
2025-09-07T08:07:07.2797518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2797585Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2797836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2797901Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2798130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2798205Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2798453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2798533Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2798793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2798868Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2799148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2799267Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2799516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2799620Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2799830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2799892Z     return self.act(input)
2025-09-07T08:07:07.2799895Z 
2025-09-07T08:07:07.2799997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2800189Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2800250Z     return mod(**inputs)
2025-09-07T08:07:07.2800496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2800555Z     outputs = self.bert(
2025-09-07T08:07:07.2800803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2800933Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2801177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2801241Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2801460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2801534Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2801775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2801854Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2802110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2802179Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2802519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2802649Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2802897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2802969Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2802972Z 
2025-09-07T08:07:07.2803216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2803407Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2803465Z     return mod(**inputs)
2025-09-07T08:07:07.2803722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2803779Z     outputs = self.bert(
2025-09-07T08:07:07.2804031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2804099Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2804341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2804405Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2804623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2804696Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2804933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2805005Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2805256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2805323Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2805568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2805631Z     self_outputs = self.self(
2025-09-07T08:07:07.2805875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2805935Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2806176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2806379Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2806383Z 
2025-09-07T08:07:07.2806473Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2806665Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2806721Z     return mod(**inputs)
2025-09-07T08:07:07.2807037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2807315Z     outputs = self.bert(
2025-09-07T08:07:07.2807573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2807644Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2807888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2807958Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2808173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2808240Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2808486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2808559Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2808905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2808967Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2809211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2809281Z     self_outputs = self.self(
2025-09-07T08:07:07.2809520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2809585Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2809825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2809888Z     self.key(current_states)
2025-09-07T08:07:07.2809899Z 
2025-09-07T08:07:07.2809993Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2810188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2810254Z     return mod(**inputs)
2025-09-07T08:07:07.2810502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2810565Z     outputs = self.bert(
2025-09-07T08:07:07.2810810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2810874Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2811123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2811186Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2811406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2811472Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2811716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2811796Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2812032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2812098Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2812337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2812398Z     self_outputs = self.self(
2025-09-07T08:07:07.2812640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2812701Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2812948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2813066Z     self.value(current_states)
2025-09-07T08:07:07.2813099Z 
2025-09-07T08:07:07.2813180Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2813272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2813459Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2813524Z     return mod(**inputs)
2025-09-07T08:07:07.2813769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2813831Z     outputs = self.bert(
2025-09-07T08:07:07.2814077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2814142Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2814389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2814453Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2814736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2814805Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2815046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2815124Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2815358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2815422Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2815661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2815723Z     self_outputs = self.self(
2025-09-07T08:07:07.2815971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2816041Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2816297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2816424Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2816427Z 
2025-09-07T08:07:07.2816530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2816721Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2816783Z     return mod(**inputs)
2025-09-07T08:07:07.2817041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2817099Z     outputs = self.bert(
2025-09-07T08:07:07.2817350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2817421Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2817742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2817818Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2818040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2818116Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2818359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2818437Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2818678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2818740Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2818988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2819181Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2819430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2819506Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2819509Z 
2025-09-07T08:07:07.2819605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2819804Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2819862Z     return mod(**inputs)
2025-09-07T08:07:07.2820115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2820173Z     outputs = self.bert(
2025-09-07T08:07:07.2820419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2820492Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2820795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2820863Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2821077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2821150Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2821391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2821467Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2821734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2821805Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2822088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2822206Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2822449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2822528Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2822531Z 
2025-09-07T08:07:07.2822622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2822816Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2822872Z     return mod(**inputs)
2025-09-07T08:07:07.2823123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2823181Z     outputs = self.bert(
2025-09-07T08:07:07.2823425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2823498Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2823740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2823810Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2824032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2824103Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2824348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2824424Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2824686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2824752Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2825067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2825208Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2825448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2825555Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2825766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2825834Z     return self.act(input)
2025-09-07T08:07:07.2825837Z 
2025-09-07T08:07:07.2825929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2826116Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2826180Z     return mod(**inputs)
2025-09-07T08:07:07.2826426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2826547Z     outputs = self.bert(
2025-09-07T08:07:07.2826797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2826863Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2827110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2827173Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2827388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2827461Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2827701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2827780Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2828039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2828109Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2828383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2828512Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2828751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2828822Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2828825Z 
2025-09-07T08:07:07.2828920Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2829105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2829163Z     return mod(**inputs)
2025-09-07T08:07:07.2829413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2829471Z     outputs = self.bert(
2025-09-07T08:07:07.2829718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2829780Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2830022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2830084Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2830304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2830369Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2830610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2830720Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2830997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2831061Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2831304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2831368Z     self_outputs = self.self(
2025-09-07T08:07:07.2831610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2831672Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2831916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T08:07:07.2832116Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:07:07.2832122Z 
2025-09-07T08:07:07.2832222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2832486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2832547Z     return mod(**inputs)
2025-09-07T08:07:07.2832802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2832861Z     outputs = self.bert(
2025-09-07T08:07:07.2833110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2833176Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2833418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2833484Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2833698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2833781Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2834027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2834099Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2834338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2834400Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2834644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2834708Z     self_outputs = self.self(
2025-09-07T08:07:07.2834946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2835007Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2835249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T08:07:07.2835323Z     self.key(current_states)
2025-09-07T08:07:07.2835326Z 
2025-09-07T08:07:07.2835418Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2835611Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2835670Z     return mod(**inputs)
2025-09-07T08:07:07.2835916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2835974Z     outputs = self.bert(
2025-09-07T08:07:07.2836218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2836289Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2836529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2836630Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2836881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2836949Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2837194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2837264Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2837507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2837566Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2837805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2837868Z     self_outputs = self.self(
2025-09-07T08:07:07.2838099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2838162Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2838464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T08:07:07.2838529Z     self.value(current_states)
2025-09-07T08:07:07.2838532Z 
2025-09-07T08:07:07.2838608Z cudagraph partition due to non gpu ops
2025-09-07T08:07:07.2838700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2838891Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2838946Z     return mod(**inputs)
2025-09-07T08:07:07.2839191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2839257Z     outputs = self.bert(
2025-09-07T08:07:07.2839500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2839569Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2839812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2839873Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2840085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2840150Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2840396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2840465Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2840705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2840764Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2841004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T08:07:07.2841070Z     self_outputs = self.self(
2025-09-07T08:07:07.2841306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2841373Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2841611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T08:07:07.2841736Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:07:07.2841739Z 
2025-09-07T08:07:07.2841834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2842018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2842078Z     return mod(**inputs)
2025-09-07T08:07:07.2842323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2842423Z     outputs = self.bert(
2025-09-07T08:07:07.2842704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2842769Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2843014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2843076Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2843291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2843361Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2843602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T08:07:07.2843678Z     self_attention_outputs = self.attention(
2025-09-07T08:07:07.2843913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:07:07.2844044Z     return func(*args, **kwargs)
2025-09-07T08:07:07.2844291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T08:07:07.2844410Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:07:07.2844653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T08:07:07.2844730Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2844733Z 
2025-09-07T08:07:07.2844826Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2845009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2845071Z     return mod(**inputs)
2025-09-07T08:07:07.2845316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2845376Z     outputs = self.bert(
2025-09-07T08:07:07.2845625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2845687Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2845932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2845994Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2846209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2846285Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2846535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2846619Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2846877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2846953Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2847241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2847357Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2847604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T08:07:07.2847676Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2847679Z 
2025-09-07T08:07:07.2847773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2847956Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2848013Z     return mod(**inputs)
2025-09-07T08:07:07.2848268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2848395Z     outputs = self.bert(
2025-09-07T08:07:07.2848647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2848714Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2848956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2849026Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2849243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2849317Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2849556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2849628Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2849948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2850016Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2850296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T08:07:07.2850399Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:07:07.2850648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T08:07:07.2850753Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:07:07.2850962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:07:07.2851032Z     return self.act(input)
2025-09-07T08:07:07.2851036Z 
2025-09-07T08:07:07.2851131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2851326Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2851385Z     return mod(**inputs)
2025-09-07T08:07:07.2851633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T08:07:07.2851699Z     outputs = self.bert(
2025-09-07T08:07:07.2851947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T08:07:07.2852019Z     encoder_outputs = self.encoder(
2025-09-07T08:07:07.2852280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T08:07:07.2852348Z     layer_outputs = layer_module(
2025-09-07T08:07:07.2852581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:07:07.2852663Z     return super().__call__(*args, **kwargs)
2025-09-07T08:07:07.2852917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T08:07:07.2852992Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:07:07.2853251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:07:07.2853318Z     return forward_fn(*input_tensors)
2025-09-07T08:07:07.2853590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T08:07:07.2853720Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:07:07.2853960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T08:07:07.2854038Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2854041Z 
2025-09-07T08:07:07.2854166Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2854393Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2854452Z     return mod(**inputs)
2025-09-07T08:07:07.2854700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T08:07:07.2854797Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:07:07.2855040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T08:07:07.2855149Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:07:07.2855387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward
2025-09-07T08:07:07.2855465Z     hidden_states = self.transform(hidden_states)
2025-09-07T08:07:07.2855707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward
2025-09-07T08:07:07.2855857Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:07:07.2855860Z 
2025-09-07T08:07:07.2855951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:07.2856136Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:07:07.2856201Z     return mod(**inputs)
2025-09-07T08:07:07.2856448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T08:07:07.2856528Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:07:07.2856776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T08:07:07.2856874Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:07:07.2857124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward
2025-09-07T08:07:07.2857209Z     hidden_states = self.decoder(hidden_states)
2025-09-07T08:07:07.2857216Z 
2025-09-07T08:07:18.3833431Z pass
2025-09-07T08:07:18.3833846Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:07:21.2322369Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:07:21.2323268Z   import pynvml  # type: ignore[import]
2025-09-07T08:07:23.2406647Z 
2025-09-07T08:07:24.6385181Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:07:24.6386710Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:07:24.6387536Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:07:24.6388187Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:07:24.6389472Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:07:24.6391467Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:07:24.6392276Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:07:24.6392955Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:07:26.5206389Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:07:26.5207091Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:07:28.3347692Z 
2025-09-07T08:07:28.3348162Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:07:28.3611749Z cpu  eval  hf_BigBird                         
2025-09-07T08:07:30.9311234Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:07:33.9687729Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:07:36.9227343Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:07:44.4860237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:44.4860800Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1897, in forward
2025-09-07T08:07:44.4861341Z     attention_mask = torch.ones(((batch_size, seq_length + past_key_values_length)), device=device)
2025-09-07T08:07:44.4861590Z 
2025-09-07T08:07:45.7682436Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:45.7683122Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2090, in torch_dynamo_resume_in__pad_to_block_size_at_2085
2025-09-07T08:07:45.7683759Z     input_ids = nn.functional.pad(input_ids, (0, padding_len), value=pad_token_id)
2025-09-07T08:07:45.7684177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:07:45.7684546Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:07:45.7684712Z 
2025-09-07T08:07:45.7684818Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:45.7685356Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2103, in torch_dynamo_resume_in__pad_to_block_size_at_2085
2025-09-07T08:07:45.7685853Z     attention_mask = nn.functional.pad(
2025-09-07T08:07:45.7686178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:07:45.7686528Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:07:45.7686683Z 
2025-09-07T08:07:45.7686781Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:45.7687292Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2106, in torch_dynamo_resume_in__pad_to_block_size_at_2085
2025-09-07T08:07:45.7687907Z     token_type_ids = nn.functional.pad(token_type_ids, (0, padding_len), value=0)  # pad with token_type_id = 0
2025-09-07T08:07:45.7688351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:07:45.7697571Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:07:45.7697769Z 
2025-09-07T08:07:48.0221758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:48.0222379Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1983, in torch_dynamo_resume_in_forward_at_1935
2025-09-07T08:07:48.0223247Z     embedding_output = self.embeddings(
2025-09-07T08:07:48.0223845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 295, in forward
2025-09-07T08:07:48.0224266Z     embeddings = self.LayerNorm(embeddings)
2025-09-07T08:07:48.0224414Z 
2025-09-07T08:07:48.0224497Z cudagraph partition due to non gpu ops
2025-09-07T08:07:48.0224716Z cudagraph partition due to non gpu ops
2025-09-07T08:07:48.0224914Z cudagraph partition due to non gpu ops
2025-09-07T08:07:48.0225138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:48.0225680Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1947, in torch_dynamo_resume_in_forward_at_1935
2025-09-07T08:07:48.0226273Z     blocked_encoder_mask, band_mask, from_mask, to_mask = self.create_masks_for_block_sparse_attn(
2025-09-07T08:07:48.0226854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2060, in create_masks_for_block_sparse_attn
2025-09-07T08:07:48.0227582Z     band_mask = create_band_mask_from_inputs(blocked_encoder_mask, blocked_encoder_mask)
2025-09-07T08:07:48.0228133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2052, in create_band_mask_from_inputs
2025-09-07T08:07:48.0228582Z     exp_blocked_to_pad = torch.cat(
2025-09-07T08:07:48.0228721Z 
2025-09-07T08:07:48.0228828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:48.0229352Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1947, in torch_dynamo_resume_in_forward_at_1935
2025-09-07T08:07:48.0229973Z     blocked_encoder_mask, band_mask, from_mask, to_mask = self.create_masks_for_block_sparse_attn(
2025-09-07T08:07:48.0230567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2060, in create_masks_for_block_sparse_attn
2025-09-07T08:07:48.0231130Z     band_mask = create_band_mask_from_inputs(blocked_encoder_mask, blocked_encoder_mask)
2025-09-07T08:07:48.0231674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2055, in create_band_mask_from_inputs
2025-09-07T08:07:48.0232219Z     band_mask = torch.einsum("blq,blk->blqk", from_blocked_mask[:, 2:-2], exp_blocked_to_pad)
2025-09-07T08:07:48.0232437Z 
2025-09-07T08:07:52.5012081Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:52.5012624Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1362, in forward
2025-09-07T08:07:52.5013080Z     band_mask = band_mask.to(hidden_states.dtype)
2025-09-07T08:07:52.5013232Z 
2025-09-07T08:07:52.5013343Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:52.5013790Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1364, in forward
2025-09-07T08:07:52.5014276Z     from_mask = from_mask.to(hidden_states.dtype)
2025-09-07T08:07:52.5014443Z 
2025-09-07T08:07:52.5014540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:52.5014966Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1366, in forward
2025-09-07T08:07:52.5015376Z     to_mask = to_mask.to(hidden_states.dtype)
2025-09-07T08:07:52.5015510Z 
2025-09-07T08:07:54.2841811Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:54.2842382Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 440, in forward
2025-09-07T08:07:54.2842805Z     self.query(hidden_states)
2025-09-07T08:07:54.2842924Z 
2025-09-07T08:07:54.2843043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:54.2845429Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 445, in forward
2025-09-07T08:07:54.2845946Z     self.key(hidden_states)
2025-09-07T08:07:54.2846068Z 
2025-09-07T08:07:54.2846170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:54.2846603Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 450, in forward
2025-09-07T08:07:54.2847002Z     self.value(hidden_states)
2025-09-07T08:07:54.2847115Z 
2025-09-07T08:07:55.2900555Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:55.2901115Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1139, in _bigbird_block_rand_mask_with_head
2025-09-07T08:07:55.2901671Z     plan_block_length = np.array(plan_from_length) // from_block_size
2025-09-07T08:07:55.2901861Z 
2025-09-07T08:07:57.6446900Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6447977Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 602, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6448648Z     first_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 0], key_layer, ndim=4)
2025-09-07T08:07:57.6449204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 494, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6449747Z     inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
2025-09-07T08:07:57.6449974Z 
2025-09-07T08:07:57.6450083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6450633Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 602, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6451248Z     first_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 0], key_layer, ndim=4)
2025-09-07T08:07:57.6451793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 494, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6452309Z     inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
2025-09-07T08:07:57.6452529Z 
2025-09-07T08:07:57.6452624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6453195Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 611, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6453803Z     first_context_layer = self.torch_bmm_nd(first_attn_weights, value_layer, ndim=4)
2025-09-07T08:07:57.6454291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6454818Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6455066Z 
2025-09-07T08:07:57.6455147Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6455365Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6455570Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6455759Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6455955Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6456150Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6456343Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6456528Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6456720Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6456912Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6457105Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6457292Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6457580Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6457859Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6458138Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6458338Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6458564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6459104Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 642, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6459724Z     second_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 1], second_key_mat, ndim=4)
2025-09-07T08:07:57.6460281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 494, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6460800Z     inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
2025-09-07T08:07:57.6461015Z 
2025-09-07T08:07:57.6461096Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6461291Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6462905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6463459Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 647, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6464028Z     to_mask.new_ones([bsz, 1, 1, n_rand_blocks * to_block_size]),
2025-09-07T08:07:57.6464199Z 
2025-09-07T08:07:57.6464303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6464841Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 653, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6465411Z     rand_mask.new_ones([bsz, n_heads, from_block_size, 4 * to_block_size]),
2025-09-07T08:07:57.6465610Z 
2025-09-07T08:07:57.6465687Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6465890Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6466116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6466643Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 698, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6467138Z     first_band_product = torch.einsum(
2025-09-07T08:07:57.6467264Z 
2025-09-07T08:07:57.6467332Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6467523Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6467726Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6467934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6468462Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 704, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6468967Z     last_band_product = torch.einsum(
2025-09-07T08:07:57.6469093Z 
2025-09-07T08:07:57.6469172Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6469370Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6469556Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6469752Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6469945Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6470136Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6470322Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6470566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6471106Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 776, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6471790Z     second_last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -2], second_last_key_mat, ndim=4)
2025-09-07T08:07:57.6472375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 494, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6473026Z     inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
2025-09-07T08:07:57.6473253Z 
2025-09-07T08:07:57.6473333Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6473538Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6473764Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6474315Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 781, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6474869Z     to_mask.new_ones([bsz, 1, 1, n_rand_blocks * to_block_size]),
2025-09-07T08:07:57.6475043Z 
2025-09-07T08:07:57.6475141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6475671Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 787, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6476327Z     rand_mask.new_ones([bsz, n_heads, from_block_size, 4 * to_block_size]),
2025-09-07T08:07:57.6476520Z 
2025-09-07T08:07:57.6476602Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6476794Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6477010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6477541Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 807, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6478157Z     last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -1], key_layer, ndim=4)
2025-09-07T08:07:57.6478703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 494, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6479219Z     inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
2025-09-07T08:07:57.6479444Z 
2025-09-07T08:07:57.6479538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6480062Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 807, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6480667Z     last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -1], key_layer, ndim=4)
2025-09-07T08:07:57.6481201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 494, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6481713Z     inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:]).transpose(1, 2)
2025-09-07T08:07:57.6481924Z 
2025-09-07T08:07:57.6482014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6482538Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 813, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6483126Z     last_context_layer = self.torch_bmm_nd(last_attn_weights, value_layer, ndim=4)
2025-09-07T08:07:57.6483616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6484128Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6484358Z 
2025-09-07T08:07:57.6484450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6484975Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 602, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6485574Z     first_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 0], key_layer, ndim=4)
2025-09-07T08:07:57.6486166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6486640Z     return torch.bmm(
2025-09-07T08:07:57.6486738Z 
2025-09-07T08:07:57.6486832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6487373Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 611, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6487988Z     first_context_layer = self.torch_bmm_nd(first_attn_weights, value_layer, ndim=4)
2025-09-07T08:07:57.6488524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6489095Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6489336Z 
2025-09-07T08:07:57.6489438Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6490083Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 574, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6490606Z     rand_attn = np.stack(rand_attn, axis=0)
2025-09-07T08:07:57.6490749Z 
2025-09-07T08:07:57.6490843Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6491380Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 577, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6491945Z     rand_attn = torch.cat([rand_attn for _ in range(batch_size)], dim=0)
2025-09-07T08:07:57.6492129Z 
2025-09-07T08:07:57.6492212Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6492433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6492959Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 620, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6493466Z     second_key_mat = torch.cat(
2025-09-07T08:07:57.6493581Z 
2025-09-07T08:07:57.6493679Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6494205Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 642, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6494824Z     second_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 1], second_key_mat, ndim=4)
2025-09-07T08:07:57.6495383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6495811Z     return torch.bmm(
2025-09-07T08:07:57.6495901Z 
2025-09-07T08:07:57.6496002Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6496543Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 643, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6497044Z     second_seq_pad = torch.cat(
2025-09-07T08:07:57.6497159Z 
2025-09-07T08:07:57.6497250Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6497855Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 579, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6498390Z     rand_mask = self._create_rand_mask_from_inputs(
2025-09-07T08:07:57.6498861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in _create_rand_mask_from_inputs
2025-09-07T08:07:57.6499395Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6499887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in <listcomp>
2025-09-07T08:07:57.6500465Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6500674Z 
2025-09-07T08:07:57.6500779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6501311Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 579, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6501825Z     rand_mask = self._create_rand_mask_from_inputs(
2025-09-07T08:07:57.6502280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in _create_rand_mask_from_inputs
2025-09-07T08:07:57.6502798Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6503449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in <listcomp>
2025-09-07T08:07:57.6504058Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6504275Z 
2025-09-07T08:07:57.6504385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6504920Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 579, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6505442Z     rand_mask = self._create_rand_mask_from_inputs(
2025-09-07T08:07:57.6505903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in _create_rand_mask_from_inputs
2025-09-07T08:07:57.6506427Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6506918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in <listcomp>
2025-09-07T08:07:57.6507406Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6507622Z 
2025-09-07T08:07:57.6507722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6508264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 579, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6508798Z     rand_mask = self._create_rand_mask_from_inputs(
2025-09-07T08:07:57.6509264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in _create_rand_mask_from_inputs
2025-09-07T08:07:57.6509787Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6510277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in <listcomp>
2025-09-07T08:07:57.6510763Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6510982Z 
2025-09-07T08:07:57.6511076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6511605Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 579, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6512123Z     rand_mask = self._create_rand_mask_from_inputs(
2025-09-07T08:07:57.6512567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 994, in _create_rand_mask_from_inputs
2025-09-07T08:07:57.6513125Z     rand_mask = torch.stack([p1[i1.flatten()] for p1, i1 in zip(to_blocked_mask, rand_attn)])
2025-09-07T08:07:57.6513331Z 
2025-09-07T08:07:57.6513416Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6513635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6514223Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 651, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6514787Z     second_rand_pad = torch.cat(
2025-09-07T08:07:57.6514906Z 
2025-09-07T08:07:57.6515007Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6515528Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 630, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6516029Z     second_value_mat = torch.cat(
2025-09-07T08:07:57.6516156Z 
2025-09-07T08:07:57.6516247Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6516771Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 665, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6517371Z     second_context_layer = self.torch_bmm_nd(second_attn_weights, second_value_mat, ndim=4)
2025-09-07T08:07:57.6517949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6518468Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6518709Z 
2025-09-07T08:07:57.6518804Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6519334Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 698, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6519848Z     first_band_product = torch.einsum(
2025-09-07T08:07:57.6519976Z 
2025-09-07T08:07:57.6520077Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6520620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 676, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6521149Z     exp_blocked_key_matrix = torch.cat(
2025-09-07T08:07:57.6521286Z 
2025-09-07T08:07:57.6521383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6521915Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 687, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6522552Z     inner_band_product = self.torch_bmm_nd_transpose(middle_query_matrix, exp_blocked_key_matrix, ndim=5)
2025-09-07T08:07:57.6523111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6523538Z     return torch.bmm(
2025-09-07T08:07:57.6523635Z 
2025-09-07T08:07:57.6523727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6524258Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 693, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6524891Z     rand_band_product = self.torch_bmm_nd_transpose(middle_query_matrix, gathered_key[:, :, 1:-1], ndim=5)
2025-09-07T08:07:57.6525440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6525858Z     return torch.bmm(
2025-09-07T08:07:57.6525954Z 
2025-09-07T08:07:57.6526046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6526570Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 704, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6527077Z     last_band_product = torch.einsum(
2025-09-07T08:07:57.6527199Z 
2025-09-07T08:07:57.6527298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6527889Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 711, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6528487Z     first_band_product += (1.0 - to_mask[:, :, :, :to_block_size].unsqueeze(3)) * attn_mask_penalty
2025-09-07T08:07:57.6528709Z 
2025-09-07T08:07:57.6528802Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6529328Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 710, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6529877Z     inner_band_product += (1.0 - band_mask) * attn_mask_penalty
2025-09-07T08:07:57.6530047Z 
2025-09-07T08:07:57.6530137Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6530659Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 713, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6531290Z     rand_band_product += (1.0 - rand_mask[:, :, 1:-1]) * attn_mask_penalty
2025-09-07T08:07:57.6531479Z 
2025-09-07T08:07:57.6531572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6532094Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 712, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6532678Z     last_band_product += (1.0 - to_mask[:, :, :, -to_block_size:].unsqueeze(3)) * attn_mask_penalty
2025-09-07T08:07:57.6532886Z 
2025-09-07T08:07:57.6532975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6533500Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 716, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6533999Z     band_product = torch.cat(
2025-09-07T08:07:57.6534111Z 
2025-09-07T08:07:57.6534219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6534748Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 679, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6535252Z     exp_blocked_value_matrix = torch.cat(
2025-09-07T08:07:57.6535390Z 
2025-09-07T08:07:57.6535482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6536006Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 727, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6536508Z     context_layer = self.torch_bmm_nd(
2025-09-07T08:07:57.6536918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6537439Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6537736Z 
2025-09-07T08:07:57.6537832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6538359Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 734, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6538861Z     context_layer += self.torch_bmm_nd(
2025-09-07T08:07:57.6539263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6539766Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6540001Z 
2025-09-07T08:07:57.6540092Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6540615Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 740, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6541246Z     context_layer += torch.einsum(
2025-09-07T08:07:57.6541372Z 
2025-09-07T08:07:57.6541478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6542022Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 743, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6542516Z     context_layer += torch.einsum(
2025-09-07T08:07:57.6542640Z 
2025-09-07T08:07:57.6542734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6543260Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 754, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6543763Z     second_last_key_mat = torch.cat(
2025-09-07T08:07:57.6543884Z 
2025-09-07T08:07:57.6543982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6544576Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 776, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6545254Z     second_last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -2], second_last_key_mat, ndim=4)
2025-09-07T08:07:57.6545830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6546257Z     return torch.bmm(
2025-09-07T08:07:57.6546348Z 
2025-09-07T08:07:57.6546448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6546966Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 777, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6547466Z     second_last_seq_pad = torch.cat(
2025-09-07T08:07:57.6547594Z 
2025-09-07T08:07:57.6547668Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6547893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6548420Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 785, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6548916Z     second_last_rand_pad = torch.cat(
2025-09-07T08:07:57.6549047Z 
2025-09-07T08:07:57.6549143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6549664Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 764, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6550166Z     second_last_value_mat = torch.cat(
2025-09-07T08:07:57.6550287Z 
2025-09-07T08:07:57.6550385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6550902Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 799, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6551529Z     second_last_context_layer = self.torch_bmm_nd(second_last_attn_weights, second_last_value_mat, ndim=4)
2025-09-07T08:07:57.6552061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6552567Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6552802Z 
2025-09-07T08:07:57.6552901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6553421Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 807, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6554019Z     last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -1], key_layer, ndim=4)
2025-09-07T08:07:57.6554640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T08:07:57.6555069Z     return torch.bmm(
2025-09-07T08:07:57.6555164Z 
2025-09-07T08:07:57.6555270Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6555797Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 813, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6556366Z     last_context_layer = self.torch_bmm_nd(last_attn_weights, value_layer, ndim=4)
2025-09-07T08:07:57.6556854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T08:07:57.6557362Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T08:07:57.6557593Z 
2025-09-07T08:07:57.6557676Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6557984Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6558198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6558725Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 743, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6559225Z     context_layer += torch.einsum(
2025-09-07T08:07:57.6559347Z 
2025-09-07T08:07:57.6559424Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6559618Z cudagraph partition due to non gpu ops
2025-09-07T08:07:57.6559827Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6560361Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 817, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6560865Z     context_layer = torch.cat(
2025-09-07T08:07:57.6560980Z 
2025-09-07T08:07:57.6561084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:07:57.6561618Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 821, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T08:07:57.6562196Z     context_layer = context_layer.view((bsz, n_heads, from_seq_len, -1)) * from_mask
2025-09-07T08:07:57.6562411Z 
2025-09-07T08:08:07.5210818Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:07.5211421Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 478, in torch_dynamo_resume_in_forward_at_455
2025-09-07T08:08:07.5212032Z     context_layer = context_layer.contiguous().view(batch_size, from_seq_length, -1)
2025-09-07T08:08:07.5212265Z 
2025-09-07T08:08:07.7235351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:07.7236012Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1385, in torch_dynamo_resume_in_forward_at_1381
2025-09-07T08:08:07.7236596Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:08:07.7237081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1295, in forward
2025-09-07T08:08:07.7237512Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:08:07.7237655Z 
2025-09-07T08:08:07.9783784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:07.9784406Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1505, in torch_dynamo_resume_in_forward_at_1468
2025-09-07T08:08:07.9784923Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:08:07.9785421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:07.9786377Z     return forward_fn(*input_tensors)
2025-09-07T08:08:07.9786922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1512, in feed_forward_chunk
2025-09-07T08:08:07.9787406Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:08:07.9787872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward
2025-09-07T08:08:07.9788317Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:08:07.9788462Z 
2025-09-07T08:08:07.9788575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:07.9789076Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1505, in torch_dynamo_resume_in_forward_at_1468
2025-09-07T08:08:07.9789558Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:08:07.9789966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:07.9790527Z     return forward_fn(*input_tensors)
2025-09-07T08:08:07.9790963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1512, in feed_forward_chunk
2025-09-07T08:08:07.9791433Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:08:07.9791882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1402, in forward
2025-09-07T08:08:07.9792322Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:08:07.9792711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:07.9793179Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:07.9793421Z 
2025-09-07T08:08:07.9793528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:07.9794030Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1505, in torch_dynamo_resume_in_forward_at_1468
2025-09-07T08:08:07.9794511Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:08:07.9794915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:07.9795316Z     return forward_fn(*input_tensors)
2025-09-07T08:08:07.9795732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1513, in feed_forward_chunk
2025-09-07T08:08:07.9796215Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:08:07.9796674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1415, in forward
2025-09-07T08:08:07.9797083Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:08:07.9797220Z 
2025-09-07T08:08:08.1999464Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:08.2000006Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 440, in forward
2025-09-07T08:08:08.2000427Z     self.query(hidden_states)
2025-09-07T08:08:08.2000546Z 
2025-09-07T08:08:08.2000660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:08.2001092Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 445, in forward
2025-09-07T08:08:08.2001491Z     self.key(hidden_states)
2025-09-07T08:08:08.2001611Z 
2025-09-07T08:08:08.2001710Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:08.2002140Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 450, in forward
2025-09-07T08:08:08.2002543Z     self.value(hidden_states)
2025-09-07T08:08:08.2002999Z 
2025-09-07T08:08:10.8497127Z W0907 08:08:10.848857 49872 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] torch._dynamo hit config.recompile_limit (8)
2025-09-07T08:08:10.8498494Z W0907 08:08:10.848857 49872 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    function: 'forward' (/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py:417)
2025-09-07T08:08:10.8500289Z W0907 08:08:10.848857 49872 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    last reason: 10/7: self.seed == 7                                           # context_layer, attention_probs = self.bigbird_block_sparse_attention(  # transformers/models/big_bird/modeling_big_bird.py:455 in forward (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
2025-09-07T08:08:10.8502085Z W0907 08:08:10.848857 49872 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-09-07T08:08:10.8502901Z W0907 08:08:10.848857 49872 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2025-09-07T08:08:11.5237863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:11.5238474Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2010, in torch_dynamo_resume_in_forward_at_1991
2025-09-07T08:08:11.5239144Z     pooler_output = self.activation(self.pooler(sequence_output[:, 0, :])) if (self.pooler is not None) else None
2025-09-07T08:08:11.5239429Z 
2025-09-07T08:08:11.5239538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:11.5240049Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2010, in torch_dynamo_resume_in_forward_at_1991
2025-09-07T08:08:11.5240701Z     pooler_output = self.activation(self.pooler(sequence_output[:, 0, :])) if (self.pooler is not None) else None
2025-09-07T08:08:11.5240967Z 
2025-09-07T08:08:11.7438611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:11.7439220Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2318, in torch_dynamo_resume_in_forward_at_2303
2025-09-07T08:08:11.7439738Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:08:11.7440188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1682, in forward
2025-09-07T08:08:11.7440645Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:08:11.7441081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1670, in forward
2025-09-07T08:08:11.7441519Z     hidden_states = self.transform(hidden_states)
2025-09-07T08:08:11.7441951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1645, in forward
2025-09-07T08:08:11.7442354Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:08:11.7442492Z 
2025-09-07T08:08:11.7442603Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:11.7443100Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2318, in torch_dynamo_resume_in_forward_at_2303
2025-09-07T08:08:11.7443578Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:08:11.7443987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1682, in forward
2025-09-07T08:08:11.7444420Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:08:11.7444844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1670, in forward
2025-09-07T08:08:11.7445775Z     hidden_states = self.transform(hidden_states)
2025-09-07T08:08:11.7446219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1645, in forward
2025-09-07T08:08:11.7446632Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:08:11.7446772Z 
2025-09-07T08:08:11.7446875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:11.7447373Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2318, in torch_dynamo_resume_in_forward_at_2303
2025-09-07T08:08:11.7447862Z     prediction_scores = self.cls(sequence_output)
2025-09-07T08:08:11.7448288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1682, in forward
2025-09-07T08:08:11.7448730Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T08:08:11.7449320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1671, in forward
2025-09-07T08:08:11.7449740Z     hidden_states = self.decoder(hidden_states)
2025-09-07T08:08:11.7449880Z 
2025-09-07T08:08:13.9147809Z pass
2025-09-07T08:08:13.9148178Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:16.8338168Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:08:16.8339286Z   import pynvml  # type: ignore[import]
2025-09-07T08:08:18.8354191Z 
2025-09-07T08:08:21.4277487Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:08:21.4277871Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:08:21.4415902Z cpu  eval  hf_DistilBert                      
2025-09-07T08:08:22.0345195Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:22.2056557Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:22.4210670Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:27.2102713Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2103010Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2103332Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2103531Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2103724Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2103908Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2104138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2104540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2104940Z     return mod(**inputs)
2025-09-07T08:08:27.2105381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2105841Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2106286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2106712Z     return self.transformer(
2025-09-07T08:08:27.2107124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2107531Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2107885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2108250Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2109015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2109537Z     sa_output = self.attention(
2025-09-07T08:08:27.2109933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T08:08:27.2110397Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T08:08:27.2110582Z 
2025-09-07T08:08:27.2110686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2111065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2111430Z     return mod(**inputs)
2025-09-07T08:08:27.2111812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2112229Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2112786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2113205Z     return self.transformer(
2025-09-07T08:08:27.2113588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2114010Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2114363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2114729Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2115146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2115558Z     sa_output = self.attention(
2025-09-07T08:08:27.2115951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T08:08:27.2116408Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2116585Z 
2025-09-07T08:08:27.2116694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2117045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2117356Z     return mod(**inputs)
2025-09-07T08:08:27.2117735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2118144Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2118539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2118937Z     return self.transformer(
2025-09-07T08:08:27.2119331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2119736Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2120079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2120434Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2120833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2121238Z     sa_output = self.attention(
2025-09-07T08:08:27.2121621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T08:08:27.2122076Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2122250Z 
2025-09-07T08:08:27.2122333Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2122684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2123031Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2123435Z     return mod(**inputs)
2025-09-07T08:08:27.2123821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2124235Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2124627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2125032Z     return self.transformer(
2025-09-07T08:08:27.2125421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2125829Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2126155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2126508Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2126989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2127405Z     sa_output = self.attention(
2025-09-07T08:08:27.2127790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T08:08:27.2128247Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:27.2128435Z 
2025-09-07T08:08:27.2128530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2128869Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2129186Z     return mod(**inputs)
2025-09-07T08:08:27.2129565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2129968Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2130367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2130786Z     return self.transformer(
2025-09-07T08:08:27.2131167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2131581Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2131909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2132261Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2132666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2133071Z     sa_output = self.attention(
2025-09-07T08:08:27.2133457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T08:08:27.2133870Z     attn_output = self.out_lin(attn_output)
2025-09-07T08:08:27.2134011Z 
2025-09-07T08:08:27.2134107Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2134448Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2134760Z     return mod(**inputs)
2025-09-07T08:08:27.2135134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2135535Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2135929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2136332Z     return self.transformer(
2025-09-07T08:08:27.2136713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2137118Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2137577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2137977Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2138386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2138828Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2139260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2139802Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2140331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2140725Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2141223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T08:08:27.2141634Z     x = self.lin1(input)
2025-09-07T08:08:27.2141739Z 
2025-09-07T08:08:27.2141837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2142191Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2142507Z     return mod(**inputs)
2025-09-07T08:08:27.2142886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2143292Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2143692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2144112Z     return self.transformer(
2025-09-07T08:08:27.2144506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2144928Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2145258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2145617Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2146031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2146476Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2146909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2147447Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2147972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2148378Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2148790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T08:08:27.2149201Z     x = self.activation(x)
2025-09-07T08:08:27.2149523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:08:27.2149858Z     return self.act(input)
2025-09-07T08:08:27.2149959Z 
2025-09-07T08:08:27.2150068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2150420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2150737Z     return mod(**inputs)
2025-09-07T08:08:27.2151121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2151584Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2152033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2152438Z     return self.transformer(
2025-09-07T08:08:27.2152830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2153240Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2153577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2153941Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2154342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2154783Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2155220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2155862Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2156384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2156772Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2157189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T08:08:27.2157595Z     x = self.lin2(x)
2025-09-07T08:08:27.2157690Z 
2025-09-07T08:08:27.2157793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2158143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2158465Z     return mod(**inputs)
2025-09-07T08:08:27.2158868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2159290Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2159704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2160110Z     return self.transformer(
2025-09-07T08:08:27.2160503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2160912Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2161259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2161620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2162027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2162440Z     sa_output = self.attention(
2025-09-07T08:08:27.2162838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T08:08:27.2163301Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T08:08:27.2163477Z 
2025-09-07T08:08:27.2163581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2163930Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2164250Z     return mod(**inputs)
2025-09-07T08:08:27.2164633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2165044Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2165440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2165906Z     return self.transformer(
2025-09-07T08:08:27.2166339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2166753Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2167096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2167449Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2167862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2168269Z     sa_output = self.attention(
2025-09-07T08:08:27.2168663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T08:08:27.2169121Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2169297Z 
2025-09-07T08:08:27.2169395Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2169828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2170149Z     return mod(**inputs)
2025-09-07T08:08:27.2170533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2170946Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2171338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2171743Z     return self.transformer(
2025-09-07T08:08:27.2172133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2172539Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2172871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2173230Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2173640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2174047Z     sa_output = self.attention(
2025-09-07T08:08:27.2174436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T08:08:27.2174884Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2175062Z 
2025-09-07T08:08:27.2175141Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2175370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2175715Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2176026Z     return mod(**inputs)
2025-09-07T08:08:27.2176408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2176821Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2177217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2177687Z     return self.transformer(
2025-09-07T08:08:27.2178080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2178491Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2178827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2179184Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2179595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2180057Z     sa_output = self.attention(
2025-09-07T08:08:27.2180491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T08:08:27.2180957Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:27.2181139Z 
2025-09-07T08:08:27.2181242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2181589Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2181904Z     return mod(**inputs)
2025-09-07T08:08:27.2182280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2182687Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2183089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2183492Z     return self.transformer(
2025-09-07T08:08:27.2183946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2184361Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2184697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2185050Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2185452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2185859Z     sa_output = self.attention(
2025-09-07T08:08:27.2186246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T08:08:27.2186665Z     attn_output = self.out_lin(attn_output)
2025-09-07T08:08:27.2186796Z 
2025-09-07T08:08:27.2186903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2187245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2187573Z     return mod(**inputs)
2025-09-07T08:08:27.2187973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2188395Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2188798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2189213Z     return self.transformer(
2025-09-07T08:08:27.2189621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2190037Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2190382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2190744Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2191165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2191613Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2192058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2192598Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2193114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2193513Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2193926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T08:08:27.2194391Z     x = self.lin1(input)
2025-09-07T08:08:27.2194532Z 
2025-09-07T08:08:27.2194644Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2194997Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2195323Z     return mod(**inputs)
2025-09-07T08:08:27.2195706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2196120Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2196515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2196921Z     return self.transformer(
2025-09-07T08:08:27.2197315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2197731Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2198147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2198509Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2198920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2199363Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2199805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2200339Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2200852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2201246Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2201659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T08:08:27.2202075Z     x = self.activation(x)
2025-09-07T08:08:27.2202389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:08:27.2202741Z     return self.act(input)
2025-09-07T08:08:27.2202850Z 
2025-09-07T08:08:27.2202947Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2203470Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2203789Z     return mod(**inputs)
2025-09-07T08:08:27.2204165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2204579Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2204977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2205393Z     return self.transformer(
2025-09-07T08:08:27.2205783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2206187Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2206525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2206878Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2207286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2207720Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2208158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2208795Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2209400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2218838Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2219382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T08:08:27.2219892Z     x = self.lin2(x)
2025-09-07T08:08:27.2220005Z 
2025-09-07T08:08:27.2220113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2220491Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2220824Z     return mod(**inputs)
2025-09-07T08:08:27.2221241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2221694Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2222294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2222726Z     return self.transformer(
2025-09-07T08:08:27.2223128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2223542Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2223885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2224251Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2224674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2225093Z     sa_output = self.attention(
2025-09-07T08:08:27.2225486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T08:08:27.2225962Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T08:08:27.2226147Z 
2025-09-07T08:08:27.2226251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2226621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2226954Z     return mod(**inputs)
2025-09-07T08:08:27.2227338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2227754Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2228161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2228581Z     return self.transformer(
2025-09-07T08:08:27.2228983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2229392Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2229740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2230109Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2230535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2230939Z     sa_output = self.attention(
2025-09-07T08:08:27.2231336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T08:08:27.2231793Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2231969Z 
2025-09-07T08:08:27.2232079Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2232436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2234305Z     return mod(**inputs)
2025-09-07T08:08:27.2234710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2235133Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2235543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2235963Z     return self.transformer(
2025-09-07T08:08:27.2236347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2236754Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2237101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2237468Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2237952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2238372Z     sa_output = self.attention(
2025-09-07T08:08:27.2238765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T08:08:27.2239226Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2239405Z 
2025-09-07T08:08:27.2239491Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2239718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2240074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2240399Z     return mod(**inputs)
2025-09-07T08:08:27.2240790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2241200Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2241602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2242011Z     return self.transformer(
2025-09-07T08:08:27.2242419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2242830Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2243165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2243526Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2243940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2244353Z     sa_output = self.attention(
2025-09-07T08:08:27.2244740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T08:08:27.2245205Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:27.2245392Z 
2025-09-07T08:08:27.2245489Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2245837Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2246157Z     return mod(**inputs)
2025-09-07T08:08:27.2246536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2246937Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2247333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2247747Z     return self.transformer(
2025-09-07T08:08:27.2248136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2248629Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2248984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2249359Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2249812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2250250Z     sa_output = self.attention(
2025-09-07T08:08:27.2250651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T08:08:27.2251075Z     attn_output = self.out_lin(attn_output)
2025-09-07T08:08:27.2251216Z 
2025-09-07T08:08:27.2251313Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2251668Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2251990Z     return mod(**inputs)
2025-09-07T08:08:27.2252443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2252866Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2253288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2253711Z     return self.transformer(
2025-09-07T08:08:27.2254105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2254522Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2254875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2255245Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2255661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2256114Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2256567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2257121Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2257703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2258111Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2258522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T08:08:27.2258935Z     x = self.lin1(input)
2025-09-07T08:08:27.2259044Z 
2025-09-07T08:08:27.2259148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2259519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2259843Z     return mod(**inputs)
2025-09-07T08:08:27.2260220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2260635Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2261041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2261452Z     return self.transformer(
2025-09-07T08:08:27.2261833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2262240Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2262584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2263006Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2263464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2263926Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2264365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2264903Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2265428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2265825Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2266271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T08:08:27.2266692Z     x = self.activation(x)
2025-09-07T08:08:27.2267087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:08:27.2267433Z     return self.act(input)
2025-09-07T08:08:27.2267533Z 
2025-09-07T08:08:27.2267639Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2267986Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2268310Z     return mod(**inputs)
2025-09-07T08:08:27.2268698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2269115Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2269510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2269920Z     return self.transformer(
2025-09-07T08:08:27.2270322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2270734Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2271076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2271428Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2271841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2272286Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2272730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2273267Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2273780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2274183Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2274596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T08:08:27.2275002Z     x = self.lin2(x)
2025-09-07T08:08:27.2275095Z 
2025-09-07T08:08:27.2275200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2275546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2275871Z     return mod(**inputs)
2025-09-07T08:08:27.2276254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2276667Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2277058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2277571Z     return self.transformer(
2025-09-07T08:08:27.2277967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2278380Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2278719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2279071Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2279481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2279888Z     sa_output = self.attention(
2025-09-07T08:08:27.2280285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T08:08:27.2280754Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T08:08:27.2280933Z 
2025-09-07T08:08:27.2281096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2281452Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2281768Z     return mod(**inputs)
2025-09-07T08:08:27.2282147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2282553Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2282949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2283356Z     return self.transformer(
2025-09-07T08:08:27.2283743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2284159Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2284503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2284862Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2285272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2285690Z     sa_output = self.attention(
2025-09-07T08:08:27.2286075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T08:08:27.2286531Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2286705Z 
2025-09-07T08:08:27.2286800Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2287149Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2287464Z     return mod(**inputs)
2025-09-07T08:08:27.2287842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2288254Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2288653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2289055Z     return self.transformer(
2025-09-07T08:08:27.2289432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2289835Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2290164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2290516Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2290919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2291366Z     sa_output = self.attention(
2025-09-07T08:08:27.2291787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T08:08:27.2292235Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2292406Z 
2025-09-07T08:08:27.2292488Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2292711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2293048Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2293360Z     return mod(**inputs)
2025-09-07T08:08:27.2293736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2294146Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2294535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2295013Z     return self.transformer(
2025-09-07T08:08:27.2295398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2295803Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2296130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2296482Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2296883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2297286Z     sa_output = self.attention(
2025-09-07T08:08:27.2297732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T08:08:27.2298199Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:27.2298384Z 
2025-09-07T08:08:27.2298481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2298834Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2299146Z     return mod(**inputs)
2025-09-07T08:08:27.2299519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2299921Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2300311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2300710Z     return self.transformer(
2025-09-07T08:08:27.2301092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2301491Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2301818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2302169Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2302565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2302967Z     sa_output = self.attention(
2025-09-07T08:08:27.2303518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T08:08:27.2303932Z     attn_output = self.out_lin(attn_output)
2025-09-07T08:08:27.2304067Z 
2025-09-07T08:08:27.2304163Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2304503Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2304816Z     return mod(**inputs)
2025-09-07T08:08:27.2305183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2305726Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2306138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2306542Z     return self.transformer(
2025-09-07T08:08:27.2306925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2307318Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2307648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2307997Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2308402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2308845Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2309380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2309925Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2310452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2310854Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2311266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T08:08:27.2311671Z     x = self.lin1(input)
2025-09-07T08:08:27.2311774Z 
2025-09-07T08:08:27.2311871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2312220Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2312535Z     return mod(**inputs)
2025-09-07T08:08:27.2312905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2313313Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2313713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2314118Z     return self.transformer(
2025-09-07T08:08:27.2314503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2314903Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2315234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2315582Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2315995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2316430Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2316858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2317393Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2317911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2318304Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2318715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T08:08:27.2319123Z     x = self.activation(x)
2025-09-07T08:08:27.2319440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:08:27.2319869Z     return self.act(input)
2025-09-07T08:08:27.2319980Z 
2025-09-07T08:08:27.2320089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2320441Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2320759Z     return mod(**inputs)
2025-09-07T08:08:27.2321139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2321549Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2321951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2322349Z     return self.transformer(
2025-09-07T08:08:27.2322735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2323144Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2323569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2323936Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2324344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2324788Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2325224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2325759Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2326272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2326662Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2327073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T08:08:27.2327476Z     x = self.lin2(x)
2025-09-07T08:08:27.2327569Z 
2025-09-07T08:08:27.2327671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2328015Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2328334Z     return mod(**inputs)
2025-09-07T08:08:27.2328712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2329117Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2329511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2329909Z     return self.transformer(
2025-09-07T08:08:27.2330296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2330705Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2331041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2331396Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2331793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2332202Z     sa_output = self.attention(
2025-09-07T08:08:27.2332593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T08:08:27.2333055Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T08:08:27.2333228Z 
2025-09-07T08:08:27.2333323Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2333720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2334071Z     return mod(**inputs)
2025-09-07T08:08:27.2334450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2334856Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2335245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2335647Z     return self.transformer(
2025-09-07T08:08:27.2336027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2336425Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2336751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2337098Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2337614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2338021Z     sa_output = self.attention(
2025-09-07T08:08:27.2338402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T08:08:27.2338842Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2339016Z 
2025-09-07T08:08:27.2339108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2339443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2339745Z     return mod(**inputs)
2025-09-07T08:08:27.2340111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2340509Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2340908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2341311Z     return self.transformer(
2025-09-07T08:08:27.2341690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2342093Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2342416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2342766Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2343169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2343572Z     sa_output = self.attention(
2025-09-07T08:08:27.2343947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T08:08:27.2344401Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2344576Z 
2025-09-07T08:08:27.2344650Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2344873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2345213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2345520Z     return mod(**inputs)
2025-09-07T08:08:27.2345896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2346301Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2346695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2347096Z     return self.transformer(
2025-09-07T08:08:27.2347531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2347982Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2348327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2348683Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2349093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2349501Z     sa_output = self.attention(
2025-09-07T08:08:27.2349895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T08:08:27.2350359Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:27.2350542Z 
2025-09-07T08:08:27.2350651Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2351070Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2351403Z     return mod(**inputs)
2025-09-07T08:08:27.2351790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2352206Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2352613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2353015Z     return self.transformer(
2025-09-07T08:08:27.2353405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2353817Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2354154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2354506Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2354932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2355344Z     sa_output = self.attention(
2025-09-07T08:08:27.2355742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T08:08:27.2356160Z     attn_output = self.out_lin(attn_output)
2025-09-07T08:08:27.2356293Z 
2025-09-07T08:08:27.2356396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2356862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2357191Z     return mod(**inputs)
2025-09-07T08:08:27.2357579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2357997Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2358396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2358809Z     return self.transformer(
2025-09-07T08:08:27.2359201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2359610Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2359946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2360305Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2360712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2361160Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2361600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2362230Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2362756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2363159Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2363584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T08:08:27.2363993Z     x = self.lin1(input)
2025-09-07T08:08:27.2364093Z 
2025-09-07T08:08:27.2364193Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2364544Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2364862Z     return mod(**inputs)
2025-09-07T08:08:27.2365243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2365733Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2366131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2366539Z     return self.transformer(
2025-09-07T08:08:27.2366923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2367334Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2367663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2368022Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2368433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2368878Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2369324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2369855Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2370373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2370768Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2371178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T08:08:27.2371593Z     x = self.activation(x)
2025-09-07T08:08:27.2371907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:08:27.2372243Z     return self.act(input)
2025-09-07T08:08:27.2372350Z 
2025-09-07T08:08:27.2372447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2372802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2373122Z     return mod(**inputs)
2025-09-07T08:08:27.2373492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2373906Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2374309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2374716Z     return self.transformer(
2025-09-07T08:08:27.2375099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2375508Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2375845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2376289Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2376704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2377144Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2377625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2378166Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2378687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2379085Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2379493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T08:08:27.2380002Z     x = self.lin2(x)
2025-09-07T08:08:27.2380106Z 
2025-09-07T08:08:27.2380205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2380559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2380883Z     return mod(**inputs)
2025-09-07T08:08:27.2381284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2381704Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2382104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2382513Z     return self.transformer(
2025-09-07T08:08:27.2382904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2383321Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2383668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2384026Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2384438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2384840Z     sa_output = self.attention(
2025-09-07T08:08:27.2385235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T08:08:27.2385692Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T08:08:27.2385866Z 
2025-09-07T08:08:27.2385970Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2386316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2386629Z     return mod(**inputs)
2025-09-07T08:08:27.2387013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2387425Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2387827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2388229Z     return self.transformer(
2025-09-07T08:08:27.2388620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2389045Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2389393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2389751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2390159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2390662Z     sa_output = self.attention(
2025-09-07T08:08:27.2391057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T08:08:27.2391519Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2391695Z 
2025-09-07T08:08:27.2391797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2392141Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2392456Z     return mod(**inputs)
2025-09-07T08:08:27.2392837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2393248Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2393645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2394136Z     return self.transformer(
2025-09-07T08:08:27.2394524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2394934Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2395269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2395620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2396025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2396432Z     sa_output = self.attention(
2025-09-07T08:08:27.2396819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T08:08:27.2397279Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T08:08:27.2397455Z 
2025-09-07T08:08:27.2397539Z cudagraph partition due to non gpu ops
2025-09-07T08:08:27.2397766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2398113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2398428Z     return mod(**inputs)
2025-09-07T08:08:27.2398798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2399209Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2399608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2400012Z     return self.transformer(
2025-09-07T08:08:27.2400399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2400803Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2401143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2401498Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2401911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2402318Z     sa_output = self.attention(
2025-09-07T08:08:27.2402699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T08:08:27.2403299Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:27.2403486Z 
2025-09-07T08:08:27.2403581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2403928Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2404316Z     return mod(**inputs)
2025-09-07T08:08:27.2404759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2405174Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2405575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2405979Z     return self.transformer(
2025-09-07T08:08:27.2406358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2406770Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2407102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2407458Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2407862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T08:08:27.2408566Z     sa_output = self.attention(
2025-09-07T08:08:27.2408959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T08:08:27.2409378Z     attn_output = self.out_lin(attn_output)
2025-09-07T08:08:27.2409512Z 
2025-09-07T08:08:27.2409611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2409959Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2410272Z     return mod(**inputs)
2025-09-07T08:08:27.2410652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2411062Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2411454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2411868Z     return self.transformer(
2025-09-07T08:08:27.2412253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2412656Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2412984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2413334Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2413727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2414176Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2414618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2415153Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2415679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2416065Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2416474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T08:08:27.2416879Z     x = self.lin1(input)
2025-09-07T08:08:27.2416977Z 
2025-09-07T08:08:27.2417074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2417470Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2417790Z     return mod(**inputs)
2025-09-07T08:08:27.2418160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2418627Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2419073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2419482Z     return self.transformer(
2025-09-07T08:08:27.2419866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2420271Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2420603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2420960Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2421361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2421796Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2422232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2422834Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2423356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2423749Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2424151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T08:08:27.2424561Z     x = self.activation(x)
2025-09-07T08:08:27.2424877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:08:27.2425206Z     return self.act(input)
2025-09-07T08:08:27.2425307Z 
2025-09-07T08:08:27.2425402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2425755Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2426068Z     return mod(**inputs)
2025-09-07T08:08:27.2426442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T08:08:27.2426856Z     dlbrt_output = self.distilbert(
2025-09-07T08:08:27.2427261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T08:08:27.2427665Z     return self.transformer(
2025-09-07T08:08:27.2428055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T08:08:27.2428462Z     layer_outputs = layer_module(
2025-09-07T08:08:27.2428793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:27.2429151Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:27.2429568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T08:08:27.2430015Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2430447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T08:08:27.2430983Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T08:08:27.2431495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:08:27.2431891Z     return forward_fn(*input_tensors)
2025-09-07T08:08:27.2432301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T08:08:27.2432702Z     x = self.lin2(x)
2025-09-07T08:08:27.2432847Z 
2025-09-07T08:08:27.2432944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2433357Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2433677Z     return mod(**inputs)
2025-09-07T08:08:27.2434054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 836, in forward
2025-09-07T08:08:27.2434557Z     prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
2025-09-07T08:08:27.2434779Z 
2025-09-07T08:08:27.2434871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:27.2435218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:27.2435530Z     return mod(**inputs)
2025-09-07T08:08:27.2435913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 839, in forward
2025-09-07T08:08:27.2436495Z     prediction_logits = self.vocab_projector(prediction_logits)  # (bs, seq_length, vocab_size)
2025-09-07T08:08:27.2436745Z 
2025-09-07T08:08:33.5649233Z pass
2025-09-07T08:08:33.5649632Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:36.0415806Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:08:36.0416783Z   import pynvml  # type: ignore[import]
2025-09-07T08:08:38.0275312Z 
2025-09-07T08:08:43.6220596Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:08:43.6221086Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:08:43.6402373Z cpu  eval  hf_GPT2                            
2025-09-07T08:08:44.1817677Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:44.3453836Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:44.4996005Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:08:51.8923736Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8924041Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8924252Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8924446Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8924643Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8924835Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8925030Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8925223Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8925417Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8925624Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8925857Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8926064Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8926327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8926713Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8927052Z     return mod(**inputs)
2025-09-07T08:08:51.8927464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8927878Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8928289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8928679Z     outputs = block(
2025-09-07T08:08:51.8929036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8929443Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8930209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8930717Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8931097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.8931558Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.8931963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8932336Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8932710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.8933210Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.8933681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.8934285Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.8934469Z 
2025-09-07T08:08:51.8934551Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8934753Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8934946Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8935136Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8935358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8935728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8936054Z     return mod(**inputs)
2025-09-07T08:08:51.8936452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8936864Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8937248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8937689Z     outputs = block(
2025-09-07T08:08:51.8938049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8938424Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8938803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8939174Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8939537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.8939940Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.8940319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8940677Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8941032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.8941436Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.8941891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.8942382Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.8942568Z 
2025-09-07T08:08:51.8942670Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8943029Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8943350Z     return mod(**inputs)
2025-09-07T08:08:51.8943720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8944124Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8944513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8944975Z     outputs = block(
2025-09-07T08:08:51.8945297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8945655Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8946020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8946394Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8946755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.8947146Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.8947520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8947889Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8948314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.8948730Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.8949178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.8949632Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.8949798Z 
2025-09-07T08:08:51.8949898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8950257Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8950580Z     return mod(**inputs)
2025-09-07T08:08:51.8950931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8951322Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8951707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8952077Z     outputs = block(
2025-09-07T08:08:51.8952402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8952758Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8953119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8953484Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8953844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.8954229Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.8954606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8955008Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8955363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.8955760Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.8956110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.8956497Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.8956670Z 
2025-09-07T08:08:51.8956766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8957140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8957458Z     return mod(**inputs)
2025-09-07T08:08:51.8957808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8958191Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8958619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8959029Z     outputs = block(
2025-09-07T08:08:51.8959365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8959711Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8960077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8960442Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8960799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.8961208Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.8961602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.8961983Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.8962423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.8962847Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.8963015Z 
2025-09-07T08:08:51.8963119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8963460Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8974354Z     return mod(**inputs)
2025-09-07T08:08:51.8974775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8975211Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8975630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8976031Z     outputs = block(
2025-09-07T08:08:51.8976376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8976749Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8977134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8977509Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8977989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.8978415Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.8978827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.8979217Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.8979562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.8980024Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.8980271Z 
2025-09-07T08:08:51.8980375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8980741Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8981060Z     return mod(**inputs)
2025-09-07T08:08:51.8981426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8981831Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8982213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8982581Z     outputs = block(
2025-09-07T08:08:51.8982892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8983328Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8983755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8984129Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8984493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.8984892Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.8985288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.8985681Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.8986045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.8986446Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.8986628Z 
2025-09-07T08:08:51.8986728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8987153Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8987482Z     return mod(**inputs)
2025-09-07T08:08:51.8987836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8988227Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8988606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8988974Z     outputs = block(
2025-09-07T08:08:51.8989289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8989650Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8990019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8990393Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8990759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T08:08:51.8991169Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T08:08:51.8991325Z 
2025-09-07T08:08:51.8991424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8991771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.8992085Z     return mod(**inputs)
2025-09-07T08:08:51.8992437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.8992827Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.8993211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.8993590Z     outputs = block(
2025-09-07T08:08:51.8993915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.8994281Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.8994654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8995027Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8995397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.8995797Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.8996180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.8996545Z     return func(*args, **kwargs)
2025-09-07T08:08:51.8996902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.8997450Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.8997970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.8998371Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.8998543Z 
2025-09-07T08:08:51.8998621Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8998824Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8999017Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8999207Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.8999430Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.8999791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9000117Z     return mod(**inputs)
2025-09-07T08:08:51.9000476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9000943Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9001327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9001703Z     outputs = block(
2025-09-07T08:08:51.9002028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9002390Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9002763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9003329Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9003695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9004093Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9004487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9004850Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9005208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9005614Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9006066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9006551Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9006734Z 
2025-09-07T08:08:51.9006835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9007185Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9007500Z     return mod(**inputs)
2025-09-07T08:08:51.9007858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9008244Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9008626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9008994Z     outputs = block(
2025-09-07T08:08:51.9009308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9009663Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9010028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9010399Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9010760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9011251Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9011685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9012060Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9012426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9012829Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9013275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9013751Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9013923Z 
2025-09-07T08:08:51.9014021Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9014378Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9014702Z     return mod(**inputs)
2025-09-07T08:08:51.9015146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9015547Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9015938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9016307Z     outputs = block(
2025-09-07T08:08:51.9016625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9016973Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9017351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9017785Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9018149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9018544Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9018917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9019289Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9019640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9020027Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9020386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9020780Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9020958Z 
2025-09-07T08:08:51.9021062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9021417Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9021745Z     return mod(**inputs)
2025-09-07T08:08:51.9022098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9022498Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9022885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9023258Z     outputs = block(
2025-09-07T08:08:51.9023574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9023925Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9024301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9024670Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9025031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9025498Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9025932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9026319Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9026670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9027060Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9027228Z 
2025-09-07T08:08:51.9027328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9027680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9027997Z     return mod(**inputs)
2025-09-07T08:08:51.9028349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9028745Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9029196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9029569Z     outputs = block(
2025-09-07T08:08:51.9029887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9030248Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9030613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9030982Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9031339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9031742Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9032141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9032521Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9032857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9033310Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9033542Z 
2025-09-07T08:08:51.9033647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9033995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9034313Z     return mod(**inputs)
2025-09-07T08:08:51.9034663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9035057Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9035437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9035808Z     outputs = block(
2025-09-07T08:08:51.9036122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9036475Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9036843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9037217Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9037577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9037973Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9038372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9038754Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9039160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9039591Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9039760Z 
2025-09-07T08:08:51.9039862Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9040201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9040524Z     return mod(**inputs)
2025-09-07T08:08:51.9040877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9041271Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9041659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9042023Z     outputs = block(
2025-09-07T08:08:51.9042336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9042757Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9043130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9043494Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9043858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9044249Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9044631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9044995Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9045340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9045833Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9046301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9046707Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9046872Z 
2025-09-07T08:08:51.9046964Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9047162Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9047359Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9047555Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9047778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9048127Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9048446Z     return mod(**inputs)
2025-09-07T08:08:51.9048799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9049188Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9049570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9049925Z     outputs = block(
2025-09-07T08:08:51.9050231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9050582Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9050954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9051315Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9051672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9052062Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9052439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9052881Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9053233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9053628Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9054079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9054560Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9054738Z 
2025-09-07T08:08:51.9054838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9055180Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9055504Z     return mod(**inputs)
2025-09-07T08:08:51.9055853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9056241Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9056672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9057043Z     outputs = block(
2025-09-07T08:08:51.9057357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9057814Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9058184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9058545Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9058902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9059288Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9059662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9060028Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9060380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9060770Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9061205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9061654Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9061811Z 
2025-09-07T08:08:51.9061901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9062235Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9062540Z     return mod(**inputs)
2025-09-07T08:08:51.9062881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9063267Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9063635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9063996Z     outputs = block(
2025-09-07T08:08:51.9064301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9064643Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9064998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9065356Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9065700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9066076Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9066510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9066959Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9067313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9067694Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9068036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9068415Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9068579Z 
2025-09-07T08:08:51.9068671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9069009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9069318Z     return mod(**inputs)
2025-09-07T08:08:51.9069668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9070124Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9070514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9070881Z     outputs = block(
2025-09-07T08:08:51.9071192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9071553Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9071940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9072297Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9072671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9073075Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9073472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9073845Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9074213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9074589Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9074753Z 
2025-09-07T08:08:51.9074849Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9075185Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9075490Z     return mod(**inputs)
2025-09-07T08:08:51.9075828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9076208Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9076587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9076948Z     outputs = block(
2025-09-07T08:08:51.9077268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9077638Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9078001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9078361Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9078706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9079150Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9079570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9079958Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9080349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9080845Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9081077Z 
2025-09-07T08:08:51.9081167Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9081522Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9081840Z     return mod(**inputs)
2025-09-07T08:08:51.9082179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9082560Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9082924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9083344Z     outputs = block(
2025-09-07T08:08:51.9083711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9084063Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9084422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9084779Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9085133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9085522Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9085915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9086293Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9086642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9087025Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9087192Z 
2025-09-07T08:08:51.9087285Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9087623Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9087927Z     return mod(**inputs)
2025-09-07T08:08:51.9088265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9088649Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9089013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9089385Z     outputs = block(
2025-09-07T08:08:51.9089717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9090066Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9090433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9090805Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9091152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T08:08:51.9091551Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T08:08:51.9091706Z 
2025-09-07T08:08:51.9091799Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9092156Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9092482Z     return mod(**inputs)
2025-09-07T08:08:51.9092844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9093221Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9093635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9094068Z     outputs = block(
2025-09-07T08:08:51.9094377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9094740Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9095114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9095476Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9095819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9096195Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9096563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9096938Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9097351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9097923Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9098395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9098791Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9098955Z 
2025-09-07T08:08:51.9099178Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9099371Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9099555Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9099738Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9099976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9100323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9100644Z     return mod(**inputs)
2025-09-07T08:08:51.9100998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9101392Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9101762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9102131Z     outputs = block(
2025-09-07T08:08:51.9102436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9102793Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9103275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9103639Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9104000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9104394Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9104769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9105136Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9105481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9105877Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9106322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9106800Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9106978Z 
2025-09-07T08:08:51.9107074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9107492Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9107870Z     return mod(**inputs)
2025-09-07T08:08:51.9108236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9108632Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9109019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9109383Z     outputs = block(
2025-09-07T08:08:51.9109691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9110047Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9110419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9110781Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9112390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9112811Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9113197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9113561Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9113923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9114326Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9114772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9115229Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9115386Z 
2025-09-07T08:08:51.9115485Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9115849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9116169Z     return mod(**inputs)
2025-09-07T08:08:51.9116525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9116915Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9117298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9117663Z     outputs = block(
2025-09-07T08:08:51.9117977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9118337Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9118706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9119077Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9119442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9119836Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9120218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9120581Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9120935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9121320Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9121677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9122067Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9122245Z 
2025-09-07T08:08:51.9122342Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9122745Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9123123Z     return mod(**inputs)
2025-09-07T08:08:51.9123476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9123860Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9124237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9124603Z     outputs = block(
2025-09-07T08:08:51.9124918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9125281Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9125646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9126018Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9126441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9126856Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9127253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9127633Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9127978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9128386Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9128552Z 
2025-09-07T08:08:51.9128655Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9128995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9129312Z     return mod(**inputs)
2025-09-07T08:08:51.9129664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9130060Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9130433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9130796Z     outputs = block(
2025-09-07T08:08:51.9131103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9131459Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9131830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9132190Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9132542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9132944Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9133348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9133730Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9134062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9134512Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9134747Z 
2025-09-07T08:08:51.9134839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9135206Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9135525Z     return mod(**inputs)
2025-09-07T08:08:51.9135867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9136319Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9136738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9137113Z     outputs = block(
2025-09-07T08:08:51.9137415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9137855Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9138235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9138608Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9138964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9139359Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9139754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9140215Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9140575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9140984Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9141157Z 
2025-09-07T08:08:51.9141252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9141595Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9141908Z     return mod(**inputs)
2025-09-07T08:08:51.9142256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9142639Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9143013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9143381Z     outputs = block(
2025-09-07T08:08:51.9143693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9144045Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9144405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9144772Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9145125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9145511Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9145883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9146248Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9146599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9147096Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9147556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9147939Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9148112Z 
2025-09-07T08:08:51.9148190Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9148390Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9148584Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9148776Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9148984Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9149334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9149651Z     return mod(**inputs)
2025-09-07T08:08:51.9150053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9150476Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9150851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9151217Z     outputs = block(
2025-09-07T08:08:51.9151529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9151885Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9152250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9152612Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9152966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9153353Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9153785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9154154Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9154511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9154913Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9155362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9155836Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9156020Z 
2025-09-07T08:08:51.9156112Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9156458Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9156777Z     return mod(**inputs)
2025-09-07T08:08:51.9157131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9157512Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9157890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9158254Z     outputs = block(
2025-09-07T08:08:51.9158563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9158909Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9159272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9159634Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9159986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9160370Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9160738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9161099Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9161446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9161837Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9162270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9162727Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9162889Z 
2025-09-07T08:08:51.9162981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9163325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9163690Z     return mod(**inputs)
2025-09-07T08:08:51.9164073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9164459Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9164837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9165202Z     outputs = block(
2025-09-07T08:08:51.9165515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9165860Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9166228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9166594Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9166948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9167398Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9167777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9168148Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9168502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9168884Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9169224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9169610Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9169786Z 
2025-09-07T08:08:51.9169879Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9170225Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9170544Z     return mod(**inputs)
2025-09-07T08:08:51.9170890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9171278Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9171656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9172023Z     outputs = block(
2025-09-07T08:08:51.9172322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9172677Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9173042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9173409Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9173762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9174177Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9174577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9174958Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9175314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9175703Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9175869Z 
2025-09-07T08:08:51.9175962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9176306Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9176618Z     return mod(**inputs)
2025-09-07T08:08:51.9176962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9177425Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9177854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9178224Z     outputs = block(
2025-09-07T08:08:51.9178529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9178880Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9179238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9179601Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9179956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9180353Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9180745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9181212Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9181558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9182008Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9182238Z 
2025-09-07T08:08:51.9182341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9182677Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9182989Z     return mod(**inputs)
2025-09-07T08:08:51.9183339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9183731Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9184108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9184470Z     outputs = block(
2025-09-07T08:08:51.9184779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9185135Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9185501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9185866Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9186218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9186615Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9187011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9187399Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9187747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9188138Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9188310Z 
2025-09-07T08:08:51.9188402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9188751Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9189065Z     return mod(**inputs)
2025-09-07T08:08:51.9189405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9189794Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9190168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9190536Z     outputs = block(
2025-09-07T08:08:51.9190882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9191283Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9191655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9192025Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9192386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T08:08:51.9192801Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T08:08:51.9192966Z 
2025-09-07T08:08:51.9193060Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9193401Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9193715Z     return mod(**inputs)
2025-09-07T08:08:51.9194058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9194508Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9194890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9195257Z     outputs = block(
2025-09-07T08:08:51.9195564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9195910Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9196288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9196665Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9197048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9197457Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9197862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9198238Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9198611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9199110Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9199576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9199971Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9200147Z 
2025-09-07T08:08:51.9200224Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9200431Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9200627Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9200816Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9201049Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9201418Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9201745Z     return mod(**inputs)
2025-09-07T08:08:51.9202101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9202502Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9202898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9203410Z     outputs = block(
2025-09-07T08:08:51.9203737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9204093Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9204470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9204917Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9205342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9205735Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9206113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9206482Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9206840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9207238Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9207680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9208164Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9208353Z 
2025-09-07T08:08:51.9208453Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9208894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9209223Z     return mod(**inputs)
2025-09-07T08:08:51.9209575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9209972Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9210356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9210724Z     outputs = block(
2025-09-07T08:08:51.9211038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9211391Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9211763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9212140Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9212502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9212882Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9213271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9213647Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9214021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9214431Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9214875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9215337Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9215507Z 
2025-09-07T08:08:51.9215610Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9215970Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9216292Z     return mod(**inputs)
2025-09-07T08:08:51.9216643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9217039Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9217425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9217861Z     outputs = block(
2025-09-07T08:08:51.9218176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9218537Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9218965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9219385Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9219752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9220143Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9220531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9220900Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9221258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9221650Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9221999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9222402Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9222585Z 
2025-09-07T08:08:51.9222750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9223114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9223430Z     return mod(**inputs)
2025-09-07T08:08:51.9223791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9224186Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9224573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9224940Z     outputs = block(
2025-09-07T08:08:51.9225247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9225605Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9225976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9226350Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9226702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9227104Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9227503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9227883Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9228232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9228613Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9228789Z 
2025-09-07T08:08:51.9228883Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9229234Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9229559Z     return mod(**inputs)
2025-09-07T08:08:51.9229915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9230304Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9230679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9231049Z     outputs = block(
2025-09-07T08:08:51.9231358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9231703Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9232071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9232437Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9232845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9233284Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9233677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9234059Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9234399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9234851Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9235085Z 
2025-09-07T08:08:51.9235199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9235543Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9235875Z     return mod(**inputs)
2025-09-07T08:08:51.9236311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9236712Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9237090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9237482Z     outputs = block(
2025-09-07T08:08:51.9237802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9238182Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9238556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9238933Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9239292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9239694Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9240117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9240499Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9240853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9241244Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9241414Z 
2025-09-07T08:08:51.9241515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9241867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9242176Z     return mod(**inputs)
2025-09-07T08:08:51.9242515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9242905Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9250281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9250679Z     outputs = block(
2025-09-07T08:08:51.9251027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9251399Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9251775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9252149Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9252528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9252961Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9253365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9253816Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9254231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9254730Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9255191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9255582Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9255762Z 
2025-09-07T08:08:51.9255841Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9256041Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9256233Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9256421Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9256637Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9256997Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9257382Z     return mod(**inputs)
2025-09-07T08:08:51.9257837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9258231Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9258616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9258984Z     outputs = block(
2025-09-07T08:08:51.9259300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9259655Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9260019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9260381Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9260743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9261129Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9261497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9261855Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9262203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9262596Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9263037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9263510Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9263695Z 
2025-09-07T08:08:51.9263797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9264149Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9264487Z     return mod(**inputs)
2025-09-07T08:08:51.9264834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9265217Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9265593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9265952Z     outputs = block(
2025-09-07T08:08:51.9266274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9266620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9266988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9267402Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9267765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9268185Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9268548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9268910Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9269256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9269643Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9270082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9270534Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9270697Z 
2025-09-07T08:08:51.9270793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9271198Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9271518Z     return mod(**inputs)
2025-09-07T08:08:51.9271865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9272248Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9272623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9272990Z     outputs = block(
2025-09-07T08:08:51.9273296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9273645Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9274014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9274378Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9274737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9275121Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9275490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9275851Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9276204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9276587Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9276932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9277322Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9277495Z 
2025-09-07T08:08:51.9277594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9277944Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9278254Z     return mod(**inputs)
2025-09-07T08:08:51.9278598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9278996Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9279368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9279731Z     outputs = block(
2025-09-07T08:08:51.9280033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9280383Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9280739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9281144Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9281536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9281931Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9282331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9282713Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9283059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9283444Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9283609Z 
2025-09-07T08:08:51.9283705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9284054Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9284368Z     return mod(**inputs)
2025-09-07T08:08:51.9284782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9285173Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9285556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9285912Z     outputs = block(
2025-09-07T08:08:51.9286211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9286553Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9286907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9287263Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9287606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9288001Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9288390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9288762Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9289092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9289534Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9289773Z 
2025-09-07T08:08:51.9289869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9290213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9290525Z     return mod(**inputs)
2025-09-07T08:08:51.9290901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9291300Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9291677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9292041Z     outputs = block(
2025-09-07T08:08:51.9292352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9292701Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9293068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9293435Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9293782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9294177Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9294662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9295305Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9295648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9296041Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9296209Z 
2025-09-07T08:08:51.9296302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9296649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9296958Z     return mod(**inputs)
2025-09-07T08:08:51.9297294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9297741Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9298111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9298544Z     outputs = block(
2025-09-07T08:08:51.9298880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9299239Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9299612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9299976Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9300334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T08:08:51.9300732Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T08:08:51.9300894Z 
2025-09-07T08:08:51.9300987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9301323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9301638Z     return mod(**inputs)
2025-09-07T08:08:51.9301979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9302355Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9302721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9303227Z     outputs = block(
2025-09-07T08:08:51.9303532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9303880Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9304245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9304612Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9304977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9305372Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9305747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9306105Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9306454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9306947Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9307407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9307797Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9307968Z 
2025-09-07T08:08:51.9308043Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9308246Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9308508Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9308744Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9308962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9309310Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9309624Z     return mod(**inputs)
2025-09-07T08:08:51.9309972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9310356Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9310729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9311087Z     outputs = block(
2025-09-07T08:08:51.9311395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9311747Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9312218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9312582Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9312934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9313312Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9313687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9314047Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9314409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9314802Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9315241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9315724Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9315904Z 
2025-09-07T08:08:51.9315998Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9316342Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9316401Z     return mod(**inputs)
2025-09-07T08:08:51.9316653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9316726Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9316967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9317025Z     outputs = block(
2025-09-07T08:08:51.9317243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9317316Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9317556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9317618Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9317859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9317934Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9318170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9318229Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9318468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9318553Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9318839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9319011Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9319015Z 
2025-09-07T08:08:51.9319106Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9319294Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9319352Z     return mod(**inputs)
2025-09-07T08:08:51.9319601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9319675Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9319916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9319972Z     outputs = block(
2025-09-07T08:08:51.9320188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9320259Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9320556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9320617Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9320861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9320935Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9321169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9321231Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9321469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9321542Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9321758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9321869Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9321873Z 
2025-09-07T08:08:51.9321964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9322150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9322213Z     return mod(**inputs)
2025-09-07T08:08:51.9322459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9322533Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9322772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9322826Z     outputs = block(
2025-09-07T08:08:51.9323042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9323115Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9323351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9323412Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9323654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9323753Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9323993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9324066Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9324277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9324385Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9324433Z 
2025-09-07T08:08:51.9324557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9324742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9324802Z     return mod(**inputs)
2025-09-07T08:08:51.9325046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9325119Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9325359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9325413Z     outputs = block(
2025-09-07T08:08:51.9325625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9325694Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9325929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9326051Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9326299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9326390Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9326632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9326706Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9326915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9327093Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9327097Z 
2025-09-07T08:08:51.9327188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9327378Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9327441Z     return mod(**inputs)
2025-09-07T08:08:51.9327688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9327761Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9328000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9328057Z     outputs = block(
2025-09-07T08:08:51.9328271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9328338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9328578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9328638Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9328882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9328973Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9329211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9329290Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9329502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9329608Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9329611Z 
2025-09-07T08:08:51.9329699Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9329882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9329941Z     return mod(**inputs)
2025-09-07T08:08:51.9330240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9330345Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9330587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9330643Z     outputs = block(
2025-09-07T08:08:51.9330857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9330923Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9331162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9331222Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9331463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9331542Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9331837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9331900Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9332140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9332324Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9332534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9332636Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9332639Z 
2025-09-07T08:08:51.9332714Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9332781Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9332850Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9332919Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9333013Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9333198Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9333255Z     return mod(**inputs)
2025-09-07T08:08:51.9333505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9333575Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9333815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9333869Z     outputs = block(
2025-09-07T08:08:51.9334081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9334150Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9334385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9334451Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9334691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9334765Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9334998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9335058Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9335297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9335384Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9335679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9335842Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9335880Z 
2025-09-07T08:08:51.9335976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9336166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9336224Z     return mod(**inputs)
2025-09-07T08:08:51.9336477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9336552Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9336794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9336852Z     outputs = block(
2025-09-07T08:08:51.9337064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9337136Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9337441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9337504Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9337815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9337897Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9338143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9338205Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9338449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9338540Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9338833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9338940Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9338947Z 
2025-09-07T08:08:51.9339044Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9339239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9339297Z     return mod(**inputs)
2025-09-07T08:08:51.9339545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9339622Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9339863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9339923Z     outputs = block(
2025-09-07T08:08:51.9340140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9340213Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9340456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9340518Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9340762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9340838Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9341072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9341132Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9341370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9341444Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9341658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9341820Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9341859Z 
2025-09-07T08:08:51.9341953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9342140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9342199Z     return mod(**inputs)
2025-09-07T08:08:51.9342446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9342521Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9342761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9342817Z     outputs = block(
2025-09-07T08:08:51.9343031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9343099Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9343403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9343465Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9343707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9343802Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9344042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9344115Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9344323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9344427Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9344431Z 
2025-09-07T08:08:51.9344520Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9344712Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9344770Z     return mod(**inputs)
2025-09-07T08:08:51.9345014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9345089Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9345327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9345383Z     outputs = block(
2025-09-07T08:08:51.9345595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9345662Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9345896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9345956Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9346202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9346290Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9346529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9346600Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9346805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9346979Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9346982Z 
2025-09-07T08:08:51.9347069Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9347254Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9347350Z     return mod(**inputs)
2025-09-07T08:08:51.9347658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9347748Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9347991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9348046Z     outputs = block(
2025-09-07T08:08:51.9348257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9348324Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9348561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9348621Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9348864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9349032Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9349275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9349355Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9349564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9349673Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9349676Z 
2025-09-07T08:08:51.9349767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9349952Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9350009Z     return mod(**inputs)
2025-09-07T08:08:51.9350255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9350332Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9350575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9350636Z     outputs = block(
2025-09-07T08:08:51.9350846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9350912Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9351147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9351206Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9351447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T08:08:51.9351546Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T08:08:51.9351550Z 
2025-09-07T08:08:51.9351641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9351829Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9351888Z     return mod(**inputs)
2025-09-07T08:08:51.9352137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9352207Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9352448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9352504Z     outputs = block(
2025-09-07T08:08:51.9352713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9352786Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9353022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9353122Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9353401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9353481Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9353721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9353784Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9354025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9354207Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9354419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9354522Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9354527Z 
2025-09-07T08:08:51.9354602Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9354733Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9354802Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9354870Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9354960Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9355145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9355205Z     return mod(**inputs)
2025-09-07T08:08:51.9355450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9355525Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9355765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9355818Z     outputs = block(
2025-09-07T08:08:51.9356033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9356105Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9356341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9356400Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9356638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9356714Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9356947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9357007Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9357245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9357336Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9357630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9357748Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9357751Z 
2025-09-07T08:08:51.9357843Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9358026Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9358083Z     return mod(**inputs)
2025-09-07T08:08:51.9358326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9358397Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9358638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9358733Z     outputs = block(
2025-09-07T08:08:51.9358949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9359048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9359285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9359343Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9359581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9359656Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9359891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9359953Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9360192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9360274Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9360624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9360723Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9360726Z 
2025-09-07T08:08:51.9360819Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9361004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9361062Z     return mod(**inputs)
2025-09-07T08:08:51.9361307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9361376Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9361621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9361676Z     outputs = block(
2025-09-07T08:08:51.9361893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9361962Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9362201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9362265Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9362509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9362587Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9362823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9362887Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9363130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9363208Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9363424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9363531Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9363534Z 
2025-09-07T08:08:51.9363627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9363815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9363873Z     return mod(**inputs)
2025-09-07T08:08:51.9364123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9364201Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9364442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9364531Z     outputs = block(
2025-09-07T08:08:51.9364778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9364848Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9365083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9365145Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9365384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9365479Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9365720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9365789Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9366003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9366163Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9366166Z 
2025-09-07T08:08:51.9366258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9366442Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9366499Z     return mod(**inputs)
2025-09-07T08:08:51.9366747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9366817Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9367062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9367116Z     outputs = block(
2025-09-07T08:08:51.9367328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9367399Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9367636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9367713Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9367951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9368040Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9368282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9368351Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9368559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9368728Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9368733Z 
2025-09-07T08:08:51.9368826Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9369008Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9369064Z     return mod(**inputs)
2025-09-07T08:08:51.9369314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9369385Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9369626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9369679Z     outputs = block(
2025-09-07T08:08:51.9369890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9369957Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9370190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9370316Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9370556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9370644Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9370889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9370969Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9371182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9371289Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9371292Z 
2025-09-07T08:08:51.9371385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9371571Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9371702Z     return mod(**inputs)
2025-09-07T08:08:51.9371953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9372028Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9372276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9372330Z     outputs = block(
2025-09-07T08:08:51.9372544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9372618Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9372855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9372919Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9373159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9373243Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9373476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9373535Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9373776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9373953Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9374166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9374268Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9374271Z 
2025-09-07T08:08:51.9374345Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9374415Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9374486Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9374555Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9374647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9374833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9374892Z     return mod(**inputs)
2025-09-07T08:08:51.9375137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9375212Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9375450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9375506Z     outputs = block(
2025-09-07T08:08:51.9375719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9375819Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9376089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9376150Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9376392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9376466Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9376698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9376759Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9376997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9377084Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9377449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9377617Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9377622Z 
2025-09-07T08:08:51.9377713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9377905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9377964Z     return mod(**inputs)
2025-09-07T08:08:51.9378223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9378296Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9378538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9378591Z     outputs = block(
2025-09-07T08:08:51.9378806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9378877Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9379114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9379172Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9379414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9379489Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9379720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9379783Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9380035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9380129Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9380429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9380534Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9380537Z 
2025-09-07T08:08:51.9380635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9380834Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9380896Z     return mod(**inputs)
2025-09-07T08:08:51.9381148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9381224Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9381474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9381530Z     outputs = block(
2025-09-07T08:08:51.9381753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9382062Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9382306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9382373Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9382614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9382696Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9382931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9382995Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9383234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9383307Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9383591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9383707Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9383710Z 
2025-09-07T08:08:51.9383808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9384003Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9384061Z     return mod(**inputs)
2025-09-07T08:08:51.9384321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9384396Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9384641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9384695Z     outputs = block(
2025-09-07T08:08:51.9384914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9384988Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9385227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9385288Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9385527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9385625Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9385865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9385934Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9386149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9386254Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9386259Z 
2025-09-07T08:08:51.9386354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9386541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9386597Z     return mod(**inputs)
2025-09-07T08:08:51.9386844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9386914Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9387155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9387210Z     outputs = block(
2025-09-07T08:08:51.9387423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9387489Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9387778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9387876Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9388116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9388210Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9388450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9388520Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9388731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9388906Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9388910Z 
2025-09-07T08:08:51.9389003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9389252Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9389312Z     return mod(**inputs)
2025-09-07T08:08:51.9389560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9389632Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9389876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9389930Z     outputs = block(
2025-09-07T08:08:51.9390144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9390211Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9390446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9390510Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9390758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9390849Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9391088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9391164Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9391377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9391481Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9391484Z 
2025-09-07T08:08:51.9391577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9391763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9391825Z     return mod(**inputs)
2025-09-07T08:08:51.9392076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9392154Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9392400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9392457Z     outputs = block(
2025-09-07T08:08:51.9392675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9392743Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9392977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9393040Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9393279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T08:08:51.9393435Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T08:08:51.9393490Z 
2025-09-07T08:08:51.9393584Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9393776Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9393833Z     return mod(**inputs)
2025-09-07T08:08:51.9394078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9394152Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9394390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9394446Z     outputs = block(
2025-09-07T08:08:51.9394661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9394727Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9395068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9395131Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9395375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9395454Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9395689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9395749Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9395988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T08:08:51.9396171Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T08:08:51.9396381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9396495Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9396498Z 
2025-09-07T08:08:51.9396572Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9396640Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9396708Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9396775Z cudagraph partition due to non gpu ops
2025-09-07T08:08:51.9396867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9397052Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9397110Z     return mod(**inputs)
2025-09-07T08:08:51.9397355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9397427Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9397671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9397731Z     outputs = block(
2025-09-07T08:08:51.9397946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9398015Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9398247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9398308Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9398547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9398622Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9398858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9398917Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9399203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9399325Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9399622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:08:51.9399740Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:08:51.9399743Z 
2025-09-07T08:08:51.9399832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9400019Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9400075Z     return mod(**inputs)
2025-09-07T08:08:51.9400324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9400393Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9400696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9400758Z     outputs = block(
2025-09-07T08:08:51.9400971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9401039Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9401272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9401333Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9401575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9401648Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9401884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9401948Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9402190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T08:08:51.9402276Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:08:51.9402564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:08:51.9402667Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:08:51.9402670Z 
2025-09-07T08:08:51.9402761Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9402946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9403002Z     return mod(**inputs)
2025-09-07T08:08:51.9403373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9403449Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9403696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9403756Z     outputs = block(
2025-09-07T08:08:51.9403967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9404035Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9404268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9404327Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9404568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T08:08:51.9404641Z     attn_output, self_attn_weights = self.attn(
2025-09-07T08:08:51.9404876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9405034Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9405327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T08:08:51.9405404Z     attn_output = self.c_proj(attn_output)
2025-09-07T08:08:51.9405617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9405722Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9405725Z 
2025-09-07T08:08:51.9405813Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9405997Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9406058Z     return mod(**inputs)
2025-09-07T08:08:51.9406303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9406376Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9406708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9406770Z     outputs = block(
2025-09-07T08:08:51.9406984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9407052Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9407293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9407352Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9407600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9407695Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9407936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T08:08:51.9408010Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T08:08:51.9408227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9408335Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9408338Z 
2025-09-07T08:08:51.9408427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9408616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9408674Z     return mod(**inputs)
2025-09-07T08:08:51.9408920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9408993Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9409232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9409296Z     outputs = block(
2025-09-07T08:08:51.9409511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9409579Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9409814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9409875Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9410118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9410207Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9410448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T08:08:51.9410524Z     hidden_states = self.act(hidden_states)
2025-09-07T08:08:51.9410728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T08:08:51.9410970Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T08:08:51.9410973Z 
2025-09-07T08:08:51.9411063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9411251Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9411308Z     return mod(**inputs)
2025-09-07T08:08:51.9411555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T08:08:51.9411628Z     transformer_outputs = self.transformer(
2025-09-07T08:08:51.9411867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T08:08:51.9411928Z     outputs = block(
2025-09-07T08:08:51.9412142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:08:51.9412213Z     return super().__call__(*args, **kwargs)
2025-09-07T08:08:51.9412509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:08:51.9412571Z     return func(*args, **kwargs)
2025-09-07T08:08:51.9412816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T08:08:51.9412905Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T08:08:51.9413142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T08:08:51.9413228Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T08:08:51.9413438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T08:08:51.9413542Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T08:08:51.9413547Z 
2025-09-07T08:08:51.9413634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:08:51.9413826Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:08:51.9413881Z     return mod(**inputs)
2025-09-07T08:08:51.9414123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward
2025-09-07T08:08:51.9414232Z     logits = self.lm_head(hidden_states[:, slice_indices, :])
2025-09-07T08:08:51.9414235Z 
2025-09-07T08:09:06.6326642Z pass
2025-09-07T08:09:06.6327068Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:09:09.2948071Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:09:09.2948984Z   import pynvml  # type: ignore[import]
2025-09-07T08:09:11.3088480Z 
2025-09-07T08:09:25.5842927Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:09:25.5843564Z loading model: 0it [00:14, ?it/s]
2025-09-07T08:09:25.6468742Z cpu  eval  hf_GPT2_large                      
2025-09-07T08:09:26.0351735Z pass_due_to_skip
2025-09-07T08:09:26.0355607Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:09:28.4600820Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:09:28.4601696Z   import pynvml  # type: ignore[import]
2025-09-07T08:09:30.4714070Z 
2025-09-07T08:09:34.4971958Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:09:34.4972331Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:09:34.5272628Z cpu  eval  hf_Longformer                      
2025-09-07T08:09:36.2713179Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:09:37.0645809Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:09:37.8520482Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:09:38.1032164Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:09:38.1032848Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:09:38.1033369Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T08:09:38.1033852Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:09:38.1034863Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T08:09:38.1035354Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:09:38.1035805Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T08:09:38.1036477Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:09:38.1037120Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(**inputs)
2025-09-07T08:09:38.1037863Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T08:09:38.1038606Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.longformer(
2025-09-07T08:09:38.1039365Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T08:09:38.1040146Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     encoder_outputs = self.encoder(
2025-09-07T08:09:38.1040911Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1244, in forward
2025-09-07T08:09:38.1041709Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2025-09-07T08:09:38.1042212Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:09:38.1042577Z W0907 08:09:38.102562 54556 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:09:44.9393034Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:09:44.9393604Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1596, in forward
2025-09-07T08:09:44.9394084Z     embedding_output = self.embeddings(
2025-09-07T08:09:44.9394543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 410, in forward
2025-09-07T08:09:44.9395104Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
2025-09-07T08:09:44.9395742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 381, in create_position_ids_from_input_ids
2025-09-07T08:09:44.9396707Z     mask = input_ids.ne(padding_idx).int()
2025-09-07T08:09:44.9396845Z 
2025-09-07T08:09:44.9396960Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:09:44.9397417Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1592, in forward
2025-09-07T08:09:44.9397978Z     extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape)[
2025-09-07T08:09:44.9398525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_utils.py", line 1763, in get_extended_attention_mask
2025-09-07T08:09:44.9399020Z     extended_attention_mask = (1.0 - extended_attention_mask) * torch.finfo(dtype).min
2025-09-07T08:09:44.9399248Z 
2025-09-07T08:09:44.9399362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:09:44.9399841Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1596, in forward
2025-09-07T08:09:44.9400406Z     embedding_output = self.embeddings(
2025-09-07T08:09:44.9400840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 410, in forward
2025-09-07T08:09:44.9401382Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
2025-09-07T08:09:44.9401992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 382, in create_position_ids_from_input_ids
2025-09-07T08:09:44.9402540Z     incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
2025-09-07T08:09:44.9402732Z 
2025-09-07T08:09:44.9402830Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:09:44.9403467Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1596, in forward
2025-09-07T08:09:44.9403902Z     embedding_output = self.embeddings(
2025-09-07T08:09:44.9404327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 410, in forward
2025-09-07T08:09:44.9404858Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
2025-09-07T08:09:44.9405446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 382, in create_position_ids_from_input_ids
2025-09-07T08:09:44.9405992Z     incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
2025-09-07T08:09:44.9406184Z 
2025-09-07T08:10:23.5105522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5106176Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5106740Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5107144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5107535Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5107972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5108409Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5108838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5109273Z     self_outputs = self.self(
2025-09-07T08:10:23.5109675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5110101Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5110245Z 
2025-09-07T08:10:23.5110349Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5111347Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5111896Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5112245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5112617Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5113037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5113464Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5113877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5114298Z     self_outputs = self.self(
2025-09-07T08:10:23.5114702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5115273Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5115787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5116382Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5116635Z 
2025-09-07T08:10:23.5116740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5117265Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5117775Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5118128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5118495Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5118916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5119338Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5119765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5120175Z     self_outputs = self.self(
2025-09-07T08:10:23.5120574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5121001Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5121137Z 
2025-09-07T08:10:23.5121238Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5121756Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5122258Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5122590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5122956Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5123367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5123782Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5124236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5124643Z     self_outputs = self.self(
2025-09-07T08:10:23.5125031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5125480Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5126070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5126637Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5127132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5127556Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5127689Z 
2025-09-07T08:10:23.5127788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5128312Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5128801Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5129130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5129558Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5129981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5130423Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5130825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5131238Z     self_outputs = self.self(
2025-09-07T08:10:23.5131636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5132109Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5132615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5133222Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5133475Z 
2025-09-07T08:10:23.5133552Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5133757Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5133955Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5134154Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5134372Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5134894Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5135380Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5135723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5136072Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5136524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5136966Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5137405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5137882Z     self_outputs = self.self(
2025-09-07T08:10:23.5146896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5147417Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5147980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5148569Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5149196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5149779Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5149985Z 
2025-09-07T08:10:23.5150068Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5150316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5150855Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5151351Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5151716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5152086Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5152516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5153023Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5153446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5153866Z     self_outputs = self.self(
2025-09-07T08:10:23.5154267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5154681Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5154800Z 
2025-09-07T08:10:23.5154910Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5155429Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5155934Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5156300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5156674Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5157150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5157586Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5158006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5158427Z     self_outputs = self.self(
2025-09-07T08:10:23.5158824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5159251Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5159373Z 
2025-09-07T08:10:23.5159475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5159987Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5160485Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5160825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5161187Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5161595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5162000Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5162410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5162819Z     self_outputs = self.self(
2025-09-07T08:10:23.5163214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5163683Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5165296Z 
2025-09-07T08:10:23.5165398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5165913Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5166408Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5166750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5167097Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5167505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5167927Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5168331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5168750Z     self_outputs = self.self(
2025-09-07T08:10:23.5169196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5169663Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5170204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5170805Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5171227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5171582Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5171737Z 
2025-09-07T08:10:23.5171834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5172356Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5172852Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5173195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5173543Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5173949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5174368Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5174777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5175198Z     self_outputs = self.self(
2025-09-07T08:10:23.5175599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5176072Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5176602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5177165Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5177750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5178214Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5178548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5178896Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5179044Z 
2025-09-07T08:10:23.5179150Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5179719Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5180239Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5180597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5180978Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5181416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5181855Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5182265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5182676Z     self_outputs = self.self(
2025-09-07T08:10:23.5183075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5183599Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5184127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5184696Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5184907Z 
2025-09-07T08:10:23.5185008Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5185523Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5186011Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5186343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5186705Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5187121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5187536Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5187942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5188347Z     self_outputs = self.self(
2025-09-07T08:10:23.5188741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5189213Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5189875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5190446Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5190655Z 
2025-09-07T08:10:23.5190758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5191278Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5191762Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5192107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5192461Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5192863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5193298Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5193711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5194248Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5194697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5195114Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5195256Z 
2025-09-07T08:10:23.5195353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5195868Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5196355Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5196687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5197033Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5197447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5197946Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5198359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5198806Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5199240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5199659Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5199799Z 
2025-09-07T08:10:23.5199891Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5200399Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5200882Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5201209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5201559Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5201967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5202388Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5202783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5203386Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5203813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5204277Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5204721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5205141Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5205278Z 
2025-09-07T08:10:23.5205376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5205892Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5206376Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5206714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5207059Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5207468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5207887Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5208368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5208819Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5209231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5209684Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5210131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5210582Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5210950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5211294Z     return self.act(input)
2025-09-07T08:10:23.5211406Z 
2025-09-07T08:10:23.5211506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5212114Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5212605Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5212938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5213297Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5213709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5214135Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5214527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5214918Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5215329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5215797Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5216252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5216665Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5216803Z 
2025-09-07T08:10:23.5216897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5217405Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5217964Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5218300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5218650Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5219067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5219486Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5219895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5220305Z     self_outputs = self.self(
2025-09-07T08:10:23.5220698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5221117Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5221254Z 
2025-09-07T08:10:23.5221346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5221859Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5222390Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5222753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5223108Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5223524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5223947Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5224348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5224759Z     self_outputs = self.self(
2025-09-07T08:10:23.5225158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5225602Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5226169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5226769Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5227020Z 
2025-09-07T08:10:23.5227117Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5227644Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5228151Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5228489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5228839Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5229255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5229671Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5230086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5230498Z     self_outputs = self.self(
2025-09-07T08:10:23.5230884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5231305Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5231444Z 
2025-09-07T08:10:23.5231540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5232055Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5232541Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5232867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5233226Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5233639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5234052Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5234459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5234858Z     self_outputs = self.self(
2025-09-07T08:10:23.5235246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5235681Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5236182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5236830Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5237361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5237773Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5237905Z 
2025-09-07T08:10:23.5238000Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5238513Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5238998Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5239325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5239675Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5240088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5240567Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5240971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5241382Z     self_outputs = self.self(
2025-09-07T08:10:23.5241767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5242206Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5242699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5243290Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5243541Z 
2025-09-07T08:10:23.5243617Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5243824Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5244027Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5244219Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5244431Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5244942Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5245439Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5245773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5246122Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5246532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5246949Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5247363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5247776Z     self_outputs = self.self(
2025-09-07T08:10:23.5248157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5248604Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5249106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5249649Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5250180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5250711Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5250967Z 
2025-09-07T08:10:23.5251100Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5251356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5251874Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5252367Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5252702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5253059Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5253472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5253884Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5254292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5254699Z     self_outputs = self.self(
2025-09-07T08:10:23.5255151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5255570Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5255687Z 
2025-09-07T08:10:23.5255786Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5256290Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5256773Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5257095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5257438Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5257914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5258331Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5258730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5259128Z     self_outputs = self.self(
2025-09-07T08:10:23.5259509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5259915Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5260041Z 
2025-09-07T08:10:23.5260135Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5260639Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5261116Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5261440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5261798Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5262194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5262601Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5263002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5263403Z     self_outputs = self.self(
2025-09-07T08:10:23.5263781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5264205Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5264341Z 
2025-09-07T08:10:23.5264641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5265138Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5265696Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5266023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5266367Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5266768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5267176Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5267571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5267971Z     self_outputs = self.self(
2025-09-07T08:10:23.5268351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5268805Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5269391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5269973Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5270391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5270728Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5270878Z 
2025-09-07T08:10:23.5270971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5271478Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5271959Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5272285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5272639Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5273047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5273456Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5273856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5274252Z     self_outputs = self.self(
2025-09-07T08:10:23.5274636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5275072Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5275596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5276144Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5276648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5277114Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5277437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5277779Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5277923Z 
2025-09-07T08:10:23.5278022Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5278534Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5279060Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5279418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5279813Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5280231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5280649Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5281065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5281476Z     self_outputs = self.self(
2025-09-07T08:10:23.5281869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5282315Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5282902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5283479Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5283685Z 
2025-09-07T08:10:23.5283784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5284296Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5284791Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5285123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5285480Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5285895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5286324Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5286745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5287161Z     self_outputs = self.self(
2025-09-07T08:10:23.5287561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5288019Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5288552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5289109Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5289323Z 
2025-09-07T08:10:23.5289424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5289945Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5290435Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5290780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5291140Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5291571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5291985Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5292398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5292847Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5293286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5293789Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5293935Z 
2025-09-07T08:10:23.5294030Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5294539Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5295023Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5295350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5295704Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5296111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5296529Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5296937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5297455Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5297940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5298358Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5298491Z 
2025-09-07T08:10:23.5298591Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5299097Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5299572Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5299899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5300251Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5300668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5301080Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5301479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5301875Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5302291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5302742Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5303320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5303743Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5303880Z 
2025-09-07T08:10:23.5303971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5304489Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5304976Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5305300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5305656Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5306065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5306488Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5306881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5307269Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5307681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5308261Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5308702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5309158Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5309531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5309872Z     return self.act(input)
2025-09-07T08:10:23.5309986Z 
2025-09-07T08:10:23.5310083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5310597Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5311076Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5311497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5311861Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5312275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5312693Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5313082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5313475Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5313882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5314351Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5314808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5315228Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5315368Z 
2025-09-07T08:10:23.5315462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5315975Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5316462Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5316794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5317140Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5317547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5317962Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5318372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5318787Z     self_outputs = self.self(
2025-09-07T08:10:23.5319175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5319598Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5319733Z 
2025-09-07T08:10:23.5319825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5320328Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5320807Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5321130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5321481Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5321933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5322385Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5322792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5323195Z     self_outputs = self.self(
2025-09-07T08:10:23.5323591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5324029Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5324533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5325119Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5325371Z 
2025-09-07T08:10:23.5325525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5326058Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5326552Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5326888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5327237Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5327646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5328062Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5328467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5328878Z     self_outputs = self.self(
2025-09-07T08:10:23.5329264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5329679Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5329812Z 
2025-09-07T08:10:23.5329905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5330413Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5330894Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5331223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5331572Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5331976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5332390Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5332808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5333216Z     self_outputs = self.self(
2025-09-07T08:10:23.5333600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5334037Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5334535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5335083Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5335582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5336043Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5336207Z 
2025-09-07T08:10:23.5336310Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5336818Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5337296Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5337686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5338050Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5338465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5338881Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5339288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5339702Z     self_outputs = self.self(
2025-09-07T08:10:23.5340163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5340615Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5341115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5341698Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5341948Z 
2025-09-07T08:10:23.5342025Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5342227Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5342424Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5342608Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5342829Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5343342Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5343830Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5344164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5344522Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5344933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5345345Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5345760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5346169Z     self_outputs = self.self(
2025-09-07T08:10:23.5346550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5347001Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5347503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5348047Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5348574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5349097Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5349305Z 
2025-09-07T08:10:23.5349379Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5349601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5350117Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5351085Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5351416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5351774Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5352188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5352624Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5353022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5353427Z     self_outputs = self.self(
2025-09-07T08:10:23.5353811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5354230Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5354348Z 
2025-09-07T08:10:23.5354528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5355032Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5355521Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5355854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5356207Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5356615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5357020Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5357422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5357832Z     self_outputs = self.self(
2025-09-07T08:10:23.5358222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5358635Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5358757Z 
2025-09-07T08:10:23.5358848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5359350Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5359830Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5360166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5360517Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5360917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5361337Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5361739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5362149Z     self_outputs = self.self(
2025-09-07T08:10:23.5362528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5362947Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5363085Z 
2025-09-07T08:10:23.5363175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5363678Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5364160Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5364482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5364907Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5365316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5365728Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5366131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5366544Z     self_outputs = self.self(
2025-09-07T08:10:23.5366949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5367412Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5367949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5368609Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5369037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5369387Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5369542Z 
2025-09-07T08:10:23.5369643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5370160Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5370648Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5370989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5371354Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5371777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5372202Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5372612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5373014Z     self_outputs = self.self(
2025-09-07T08:10:23.5373407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5373866Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5374394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5374939Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5375447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5375917Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5376251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5376595Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5376741Z 
2025-09-07T08:10:23.5376842Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5377353Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5377928Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5378273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5378636Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5379102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5379568Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5379979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5380393Z     self_outputs = self.self(
2025-09-07T08:10:23.5380786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5381232Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5381758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5382326Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5382532Z 
2025-09-07T08:10:23.5382638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5383214Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5383695Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5384030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5384387Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5384799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5385217Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5385618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5386028Z     self_outputs = self.self(
2025-09-07T08:10:23.5386420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5386872Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5387394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5387943Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5388148Z 
2025-09-07T08:10:23.5388241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5388748Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5389231Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5389563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5389916Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5390325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5390738Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5391141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5391590Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5392025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5392453Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5392595Z 
2025-09-07T08:10:23.5392690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5393256Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5393778Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5394104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5394461Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5394870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5395283Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5395682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5396121Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5396556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5397048Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5397180Z 
2025-09-07T08:10:23.5397280Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5397785Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5398270Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5398602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5398958Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5399367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5399784Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5400188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5400586Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5401004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5401474Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5401924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5402351Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5402491Z 
2025-09-07T08:10:23.5402586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5403224Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5403716Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5404054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5404413Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5404824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5405245Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5405635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5406033Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5406444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5406898Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5407336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5407917Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5408299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5408642Z     return self.act(input)
2025-09-07T08:10:23.5408752Z 
2025-09-07T08:10:23.5408859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5409372Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5409854Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5410193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5410554Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5411076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5411512Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5411912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5412308Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5412720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5413187Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5413647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5414063Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5414203Z 
2025-09-07T08:10:23.5414299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5414817Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5415308Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5415639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5415999Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5416412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5416832Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5417246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5417705Z     self_outputs = self.self(
2025-09-07T08:10:23.5418104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5418532Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5418664Z 
2025-09-07T08:10:23.5418767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5419274Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5419757Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5420098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5420452Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5420869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5421290Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5421751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5422198Z     self_outputs = self.self(
2025-09-07T08:10:23.5422596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5423035Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5423537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5424135Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5424384Z 
2025-09-07T08:10:23.5424484Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5425001Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5425555Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5425898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5426251Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5426664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5427080Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5427490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5427896Z     self_outputs = self.self(
2025-09-07T08:10:23.5428283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5428702Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5428834Z 
2025-09-07T08:10:23.5428941Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5429448Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5429925Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5430263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5430620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5431035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5431445Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5431845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5432255Z     self_outputs = self.self(
2025-09-07T08:10:23.5432643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5433084Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5433583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5434133Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5434627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5435038Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5435161Z 
2025-09-07T08:10:23.5435260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5435764Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5436346Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5436682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5437053Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5437465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5437881Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5438283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5438691Z     self_outputs = self.self(
2025-09-07T08:10:23.5439079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5439519Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5440079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5440673Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5440918Z 
2025-09-07T08:10:23.5440992Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5441186Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5441379Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5441564Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5441776Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5442275Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5442758Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5443084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5443428Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5443834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5444243Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5453681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5454140Z     self_outputs = self.self(
2025-09-07T08:10:23.5454570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5455057Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5455605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5456174Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5456719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5457248Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5457460Z 
2025-09-07T08:10:23.5457537Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5457861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5458376Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5458867Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5459207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5459715Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5460141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5460575Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5460995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5461403Z     self_outputs = self.self(
2025-09-07T08:10:23.5461801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5462206Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5462320Z 
2025-09-07T08:10:23.5462422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5463036Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5463529Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5463870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5464222Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5464642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5465052Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5465452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5465857Z     self_outputs = self.self(
2025-09-07T08:10:23.5466251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5466662Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5466785Z 
2025-09-07T08:10:23.5466884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5467390Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5467872Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5468203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5468563Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5468967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5469382Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5469783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5470200Z     self_outputs = self.self(
2025-09-07T08:10:23.5470583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5471009Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5471151Z 
2025-09-07T08:10:23.5471243Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5471749Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5472231Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5472553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5472908Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5473317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5473826Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5474226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5474628Z     self_outputs = self.self(
2025-09-07T08:10:23.5475009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5475464Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5475995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5476593Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5477012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5477424Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5477583Z 
2025-09-07T08:10:23.5477679Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5478192Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5478684Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5479014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5479363Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5479771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5480182Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5480590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5480997Z     self_outputs = self.self(
2025-09-07T08:10:23.5481384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5481825Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5482342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5482879Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5483381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5483838Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5484156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5484493Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5484635Z 
2025-09-07T08:10:23.5484728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5485230Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5485713Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5486043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5486395Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5486798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5487207Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5487657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5488107Z     self_outputs = self.self(
2025-09-07T08:10:23.5488489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5488932Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5489448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5490002Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5490202Z 
2025-09-07T08:10:23.5490295Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5490794Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5491266Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5491650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5492003Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5492406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5492808Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5493203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5493602Z     self_outputs = self.self(
2025-09-07T08:10:23.5493981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5494419Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5494937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5495486Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5495684Z 
2025-09-07T08:10:23.5495773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5496274Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5496751Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5497077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5497420Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5497862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5498282Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5498679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5499112Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5499543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5499956Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5500089Z 
2025-09-07T08:10:23.5500177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5500671Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5501144Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5501532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5501913Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5502310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5502711Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5503293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5503727Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5504159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5504566Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5504693Z 
2025-09-07T08:10:23.5504783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5505440Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5505923Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5506250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5506594Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5506990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5507397Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5507790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5508176Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5508580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5509037Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5509465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5509878Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5510007Z 
2025-09-07T08:10:23.5510106Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5510692Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5511178Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5511510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5511859Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5512266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5512682Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5513075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5513463Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5513867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5514314Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5514757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5515212Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5515583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5516060Z     return self.act(input)
2025-09-07T08:10:23.5516164Z 
2025-09-07T08:10:23.5516262Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5516769Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5517265Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5517599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5517946Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5518349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5518760Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5519145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5519605Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5520007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5520461Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5520914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5521322Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5521452Z 
2025-09-07T08:10:23.5521549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5522050Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5522525Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5522852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5523198Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5523598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5524005Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5524406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5524803Z     self_outputs = self.self(
2025-09-07T08:10:23.5525191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5525599Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5525724Z 
2025-09-07T08:10:23.5525816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5526359Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5526831Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5527152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5527492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5527886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5528290Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5528682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5529078Z     self_outputs = self.self(
2025-09-07T08:10:23.5529454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5529977Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5530480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5531066Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5531308Z 
2025-09-07T08:10:23.5531402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5531903Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5532375Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5532699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5533046Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5533711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5534129Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5534525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5534928Z     self_outputs = self.self(
2025-09-07T08:10:23.5535313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5535721Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5535846Z 
2025-09-07T08:10:23.5535939Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5536438Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5536917Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5537238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5537579Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5538052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5538457Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5538853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5539250Z     self_outputs = self.self(
2025-09-07T08:10:23.5539625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5540046Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5540542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5541084Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5541568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5541968Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5542087Z 
2025-09-07T08:10:23.5542177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5542674Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5543144Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5543467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5543910Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5544310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5544711Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5545103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5545499Z     self_outputs = self.self(
2025-09-07T08:10:23.5545875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5546297Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5546781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5547430Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5547676Z 
2025-09-07T08:10:23.5547756Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5547953Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5548134Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5548315Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5548525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5549034Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5549510Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5549840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5550189Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5550600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5551008Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5551408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5551818Z     self_outputs = self.self(
2025-09-07T08:10:23.5552209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5552309Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5552656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5552790Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5553118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5553258Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5553261Z 
2025-09-07T08:10:23.5553332Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5553429Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5553787Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5553851Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5554068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5554145Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5554424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5554581Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5554860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5554921Z     self_outputs = self.self(
2025-09-07T08:10:23.5555202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5555265Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5555268Z 
2025-09-07T08:10:23.5555358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5555711Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5555773Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5555988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5556118Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5556404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5556471Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5556748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5556811Z     self_outputs = self.self(
2025-09-07T08:10:23.5557085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5557157Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5557160Z 
2025-09-07T08:10:23.5557253Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5557605Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5557674Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5557888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5557959Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5558236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5558301Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5558577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5558635Z     self_outputs = self.self(
2025-09-07T08:10:23.5558912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5558988Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5558995Z 
2025-09-07T08:10:23.5559086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5559434Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5559497Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5559708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5559774Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5560053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5560119Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5560394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5560520Z     self_outputs = self.self(
2025-09-07T08:10:23.5560797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5560909Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5561268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5561435Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5561623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5561715Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5561718Z 
2025-09-07T08:10:23.5561809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5562252Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5562320Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5562535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5562604Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5562880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5562945Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5563223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5563283Z     self_outputs = self.self(
2025-09-07T08:10:23.5563559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5563672Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5564027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5564151Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5564469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5564552Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5564734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5564819Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5564823Z 
2025-09-07T08:10:23.5564913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5565270Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5565333Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5565546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5565614Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5565892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5565958Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5566234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5566293Z     self_outputs = self.self(
2025-09-07T08:10:23.5566568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5566735Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5567090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5567234Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5567238Z 
2025-09-07T08:10:23.5567328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5567679Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5567740Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5567958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5568024Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5568368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5568435Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5568710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5568772Z     self_outputs = self.self(
2025-09-07T08:10:23.5569047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5569151Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5569500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5569638Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5569643Z 
2025-09-07T08:10:23.5569734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5570083Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5570148Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5570358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5570431Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5570709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5570776Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5571051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5571159Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5571437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5571510Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5571513Z 
2025-09-07T08:10:23.5571605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5571953Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5572015Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5572225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5572288Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5572565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5572700Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5572981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5573077Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5573353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5573425Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5573428Z 
2025-09-07T08:10:23.5573516Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5573867Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5573929Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5574199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5574268Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5574543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5574618Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5574880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5574949Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5575231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5575332Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5575610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5575685Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5575688Z 
2025-09-07T08:10:23.5575779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5576129Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5576192Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5576402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5576466Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5576743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5576813Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5577074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5577139Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5577420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5577511Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5577848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5577953Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5578162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5578225Z     return self.act(input)
2025-09-07T08:10:23.5578228Z 
2025-09-07T08:10:23.5578317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5578707Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5578806Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5579019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5579089Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5579364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5579436Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5579691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5579754Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5580034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5580219Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5580503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5580573Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5580576Z 
2025-09-07T08:10:23.5580665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5581015Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5581077Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5581290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5581354Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5581637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5581706Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5581981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5582048Z     self_outputs = self.self(
2025-09-07T08:10:23.5582324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5582397Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5582400Z 
2025-09-07T08:10:23.5582487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5582837Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5582899Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5583112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5583184Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5583467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5583542Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5583818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5583878Z     self_outputs = self.self(
2025-09-07T08:10:23.5584159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5584253Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5584606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5584856Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5584859Z 
2025-09-07T08:10:23.5584958Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5585308Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5585371Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5585590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5585657Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5585943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5586012Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5586364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5586432Z     self_outputs = self.self(
2025-09-07T08:10:23.5586714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5586791Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5586794Z 
2025-09-07T08:10:23.5586882Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5587233Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5587296Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5587509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5587584Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5587863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5587934Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5588210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5588273Z     self_outputs = self.self(
2025-09-07T08:10:23.5588553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5588644Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5588993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5589145Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5589430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5589498Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5589501Z 
2025-09-07T08:10:23.5589589Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5589948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5590011Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5590225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5590292Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5590575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5590714Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5590988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5591056Z     self_outputs = self.self(
2025-09-07T08:10:23.5591331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5591425Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5591767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5591941Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5591951Z 
2025-09-07T08:10:23.5592026Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5592095Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5592230Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5592301Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5592396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5592747Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5592810Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5593027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5593094Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5593376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5593444Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5593726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5593795Z     self_outputs = self.self(
2025-09-07T08:10:23.5594069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5594175Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5594513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5594655Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5594981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5595119Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5595124Z 
2025-09-07T08:10:23.5595200Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5595293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5595646Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5595709Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5595919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5595993Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5596268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5596339Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5596613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5596751Z     self_outputs = self.self(
2025-09-07T08:10:23.5597030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5597096Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5597099Z 
2025-09-07T08:10:23.5597193Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5597544Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5597611Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5597826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5597896Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5598172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5598311Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5598596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5598660Z     self_outputs = self.self(
2025-09-07T08:10:23.5598940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5599010Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5599013Z 
2025-09-07T08:10:23.5599105Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5599462Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5599525Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5599740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5599815Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5600100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5600164Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5600440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5600512Z     self_outputs = self.self(
2025-09-07T08:10:23.5600788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5600864Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5600867Z 
2025-09-07T08:10:23.5600956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5601309Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5601379Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5601598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5601680Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5601968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5602048Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5602328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5602394Z     self_outputs = self.self(
2025-09-07T08:10:23.5602677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5602831Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5603384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5603558Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5603747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5603848Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5603852Z 
2025-09-07T08:10:23.5603953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5604319Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5604388Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5604738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5604814Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5605097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5605172Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5605447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5605517Z     self_outputs = self.self(
2025-09-07T08:10:23.5605797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5605909Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5606265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5606400Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5606721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5606804Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5606994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5607082Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5607086Z 
2025-09-07T08:10:23.5607182Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5607540Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5607608Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5607836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5607907Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5608190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5608259Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5608537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5608604Z     self_outputs = self.self(
2025-09-07T08:10:23.5608878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5608988Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5609345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5609640Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5609644Z 
2025-09-07T08:10:23.5609738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5610093Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5610163Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5610381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5610453Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5610731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5610798Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5611177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5611241Z     self_outputs = self.self(
2025-09-07T08:10:23.5611526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5611628Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5611987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5612122Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5612126Z 
2025-09-07T08:10:23.5612219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5612579Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5612643Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5612863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5612931Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5613213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5613280Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5613559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5613666Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5613942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5614023Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5614029Z 
2025-09-07T08:10:23.5614117Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5614464Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5614528Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5614738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5614804Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5615078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5615143Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5615417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5615585Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5615866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5615938Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5615943Z 
2025-09-07T08:10:23.5616032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5616379Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5616442Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5616654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5616717Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5617058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5617133Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5617397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5617465Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5617795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5617900Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5618179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5618252Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5618255Z 
2025-09-07T08:10:23.5618344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5618702Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5618764Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5618975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5619044Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5619319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5619392Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5619647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5619713Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5619999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5620093Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5620370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5620472Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5620682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5620745Z     return self.act(input)
2025-09-07T08:10:23.5620749Z 
2025-09-07T08:10:23.5620836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5621188Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5621249Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5621509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5621609Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5621889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5621963Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5622216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5622282Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5622562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5622676Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5622951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5623088Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5623092Z 
2025-09-07T08:10:23.5623184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5623535Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5623600Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5623818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5623890Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5624165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5624232Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5624513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5624579Z     self_outputs = self.self(
2025-09-07T08:10:23.5624857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5624928Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5624931Z 
2025-09-07T08:10:23.5625018Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5625368Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5625429Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5625642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5625706Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5625987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5626054Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5626328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5626390Z     self_outputs = self.self(
2025-09-07T08:10:23.5626662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5626756Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5627098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5627279Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5627319Z 
2025-09-07T08:10:23.5627442Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5627794Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5627860Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5628069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5628136Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5628412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5628477Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5628753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5628817Z     self_outputs = self.self(
2025-09-07T08:10:23.5629153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5629226Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5629229Z 
2025-09-07T08:10:23.5629320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5629669Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5629729Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5629941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5630006Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5630282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5630347Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5630626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5630690Z     self_outputs = self.self(
2025-09-07T08:10:23.5630964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5631057Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5631397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5631542Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5631820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5631888Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5631892Z 
2025-09-07T08:10:23.5631986Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5632334Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5632398Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5632611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5632681Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5632958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5633022Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5633298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5633411Z     self_outputs = self.self(
2025-09-07T08:10:23.5633724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5633814Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5634154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5634332Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5634335Z 
2025-09-07T08:10:23.5634408Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5634478Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5634544Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5634612Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5634701Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5635114Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5635183Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5635393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5635461Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5635736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5635801Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5636084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5636147Z     self_outputs = self.self(
2025-09-07T08:10:23.5636426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5636534Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5636870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5637007Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5637330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5637470Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5637473Z 
2025-09-07T08:10:23.5637542Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5637634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5637985Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5638049Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5638261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5638327Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5638604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5638669Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5638944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5639003Z     self_outputs = self.self(
2025-09-07T08:10:23.5639276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5639379Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5639410Z 
2025-09-07T08:10:23.5639503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5639854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5639913Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5640124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5640191Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5640470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5640541Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5640818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5640891Z     self_outputs = self.self(
2025-09-07T08:10:23.5641226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5641301Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5641304Z 
2025-09-07T08:10:23.5641400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5641749Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5641818Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5642030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5642097Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5642381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5642453Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5642736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5642798Z     self_outputs = self.self(
2025-09-07T08:10:23.5643082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5643159Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5643162Z 
2025-09-07T08:10:23.5643255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5643614Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5643679Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5643893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5643965Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5644251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5644318Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5644594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5644661Z     self_outputs = self.self(
2025-09-07T08:10:23.5644935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5645054Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5645408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5645642Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5645840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5645931Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5645935Z 
2025-09-07T08:10:23.5646031Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5646386Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5646452Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5646665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5646733Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5647076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5647146Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5647437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5647500Z     self_outputs = self.self(
2025-09-07T08:10:23.5647779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5647888Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5648239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5648375Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5648698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5648786Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5648968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5649052Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5649055Z 
2025-09-07T08:10:23.5649151Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5649498Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5649568Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5649779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5649847Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5650129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5650198Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5650482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5650544Z     self_outputs = self.self(
2025-09-07T08:10:23.5650825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5650925Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5651277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5651426Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5651465Z 
2025-09-07T08:10:23.5651556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5651952Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5652016Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5652234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5652300Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5652577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5652645Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5652922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5652991Z     self_outputs = self.self(
2025-09-07T08:10:23.5653346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5653454Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5653805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5653941Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5653945Z 
2025-09-07T08:10:23.5654038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5654393Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5654457Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5654670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5654741Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5655022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5655089Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5655370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5655472Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5655748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5655822Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5655825Z 
2025-09-07T08:10:23.5655912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5656266Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5656333Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5656562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5656635Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5656916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5656985Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5657261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5657365Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5657675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5657844Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5657847Z 
2025-09-07T08:10:23.5657936Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5658286Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5658349Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5658562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5658635Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5658911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5658990Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5659317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5659389Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5659682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5659786Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5660068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5660139Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5660142Z 
2025-09-07T08:10:23.5660239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5660594Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5660658Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5660881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5660950Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5661233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5661305Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5661566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5661640Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5661921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5662023Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5662302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5662410Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5662616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5662680Z     return self.act(input)
2025-09-07T08:10:23.5662683Z 
2025-09-07T08:10:23.5662774Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5663124Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5663190Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5663399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5663466Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5663788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5663894Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5664154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5664222Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5664509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5664624Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5664903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5664982Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5664986Z 
2025-09-07T08:10:23.5665076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5665499Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5665569Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5665784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5665858Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5666132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5666200Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5666476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5666541Z     self_outputs = self.self(
2025-09-07T08:10:23.5666816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5666894Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5666897Z 
2025-09-07T08:10:23.5666994Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5667341Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5667410Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5667621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5667692Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5667971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5668038Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5668326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5668389Z     self_outputs = self.self(
2025-09-07T08:10:23.5668669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5668760Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5669103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5669287Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5669290Z 
2025-09-07T08:10:23.5669380Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5669733Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5669863Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5670081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5670147Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5670423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5670492Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5670766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5670832Z     self_outputs = self.self(
2025-09-07T08:10:23.5671105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5671175Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5671183Z 
2025-09-07T08:10:23.5671335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5671689Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5671755Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5671969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5672037Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5672313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5672376Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5672653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5672718Z     self_outputs = self.self(
2025-09-07T08:10:23.5673007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5673098Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5673447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5673593Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5673872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5673952Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5673956Z 
2025-09-07T08:10:23.5674045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5674398Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5674466Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5674679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5674754Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5675033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5675107Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5675383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5675448Z     self_outputs = self.self(
2025-09-07T08:10:23.5675722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5675844Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5676220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5676395Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5676398Z 
2025-09-07T08:10:23.5676473Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5676541Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5676611Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5676686Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5676775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5677134Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5677202Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5677492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5677565Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5677843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5677913Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5678190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5678261Z     self_outputs = self.self(
2025-09-07T08:10:23.5678537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5678637Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5678982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5679119Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5679448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5679590Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5679593Z 
2025-09-07T08:10:23.5679671Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5679758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5680107Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5680173Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5680384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5680465Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5680749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5680826Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5681106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5681167Z     self_outputs = self.self(
2025-09-07T08:10:23.5681448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5681512Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5681515Z 
2025-09-07T08:10:23.5681605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5681955Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5682083Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5682301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5682369Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5682656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5682723Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5683009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5683072Z     self_outputs = self.self(
2025-09-07T08:10:23.5683349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5683435Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5683440Z 
2025-09-07T08:10:23.5683590Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5683952Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5684016Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5684229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5684300Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5684579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5684649Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5684924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5684996Z     self_outputs = self.self(
2025-09-07T08:10:23.5685276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5685353Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5685356Z 
2025-09-07T08:10:23.5685455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5685808Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5685876Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5686088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5686155Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5686440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5686512Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5686802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5686864Z     self_outputs = self.self(
2025-09-07T08:10:23.5687145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5687252Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5687608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5687778Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5687965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5688123Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5688126Z 
2025-09-07T08:10:23.5688216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5688566Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5688625Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5688835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5688903Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5689178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5689243Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5689521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5689644Z     self_outputs = self.self(
2025-09-07T08:10:23.5689922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5690027Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5690388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5690514Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5690837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5690918Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5691101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5691194Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5691197Z 
2025-09-07T08:10:23.5691287Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5691639Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5691700Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5691918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5691982Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5692261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5692334Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5692610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5692677Z     self_outputs = self.self(
2025-09-07T08:10:23.5692951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5693050Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5693401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5693541Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5693544Z 
2025-09-07T08:10:23.5693633Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5693983Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5694124Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5694339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5694407Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5694685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5694750Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5695030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5695092Z     self_outputs = self.self(
2025-09-07T08:10:23.5695370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5695477Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5695887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5696032Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5696035Z 
2025-09-07T08:10:23.5696125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5696483Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5696543Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5696757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5696830Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5697107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5697187Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5697464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5697575Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5697908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5697985Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5697988Z 
2025-09-07T08:10:23.5698084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5698433Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5698503Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5698723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5698793Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5699078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5699148Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5699433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5699530Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5699811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5699882Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5699885Z 
2025-09-07T08:10:23.5699970Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5700368Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5700484Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5700697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5700762Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5701043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5701114Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5701373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5701441Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5701724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5701892Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5702174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5702250Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5702260Z 
2025-09-07T08:10:23.5702354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5702709Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5702784Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5703005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5703208Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5703512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5703592Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5703871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5703941Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5704238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5704342Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5704624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5704738Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5704948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5705025Z     return self.act(input)
2025-09-07T08:10:23.5705028Z 
2025-09-07T08:10:23.5705127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5705493Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5705563Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5705785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5705861Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5706140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5706224Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5706483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5706722Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5707016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5707132Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5707417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5707492Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5707495Z 
2025-09-07T08:10:23.5707596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5707948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5708016Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5708338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5708415Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5708703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5708774Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5709058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5709126Z     self_outputs = self.self(
2025-09-07T08:10:23.5709405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5709485Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5709488Z 
2025-09-07T08:10:23.5709581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5709944Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5710007Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5710220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5710292Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5710569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5710641Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5710917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5710983Z     self_outputs = self.self(
2025-09-07T08:10:23.5711264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5711363Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5711714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5711898Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5711901Z 
2025-09-07T08:10:23.5712003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5712351Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5712414Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5712624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5712729Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5713047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5713114Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5713393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5713456Z     self_outputs = self.self(
2025-09-07T08:10:23.5713731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5713805Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5713809Z 
2025-09-07T08:10:23.5713900Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5714251Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5714376Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5714595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5714663Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5714941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5715008Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5715300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5715366Z     self_outputs = self.self(
2025-09-07T08:10:23.5715643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5715734Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5716084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5716231Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5716512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5716576Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5716580Z 
2025-09-07T08:10:23.5716671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5717020Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5717082Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5717296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5717370Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5717656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5717724Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5718009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5718070Z     self_outputs = self.self(
2025-09-07T08:10:23.5718349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5718444Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5718784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5719010Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5719043Z 
2025-09-07T08:10:23.5719121Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5719192Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5719271Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5719343Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5719444Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5719806Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5719879Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5720091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5720159Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5720507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5720580Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5720867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5720933Z     self_outputs = self.self(
2025-09-07T08:10:23.5721211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5721319Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5721661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5721810Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5722141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5722288Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5722292Z 
2025-09-07T08:10:23.5722366Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5722461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5722821Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5722889Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5723110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5723179Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5723459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5723539Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5723817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5723887Z     self_outputs = self.self(
2025-09-07T08:10:23.5724163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5724234Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5724237Z 
2025-09-07T08:10:23.5731451Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5731902Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5731976Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5732224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5732435Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5732737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5732814Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5733097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5733166Z     self_outputs = self.self(
2025-09-07T08:10:23.5733448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5733523Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5733527Z 
2025-09-07T08:10:23.5733632Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5734059Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5734138Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5734368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5734447Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5734730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5734799Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5735080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5735146Z     self_outputs = self.self(
2025-09-07T08:10:23.5735426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5735508Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5735516Z 
2025-09-07T08:10:23.5735616Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5735972Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5736036Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5736256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5736326Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5736607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5736673Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5736947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5737016Z     self_outputs = self.self(
2025-09-07T08:10:23.5737295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5737414Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5737841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5738023Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5738213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5738306Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5738309Z 
2025-09-07T08:10:23.5738411Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5738809Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5738908Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5739125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5739198Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5739474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5739543Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5739821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5739883Z     self_outputs = self.self(
2025-09-07T08:10:23.5740166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5740337Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5744804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5744974Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5745328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5745419Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5745622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5745717Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5745721Z 
2025-09-07T08:10:23.5745827Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5746201Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5746278Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5746531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5746608Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5746899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5746971Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5747255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5747325Z     self_outputs = self.self(
2025-09-07T08:10:23.5747608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5747735Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5748101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5748258Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5748262Z 
2025-09-07T08:10:23.5748362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5748723Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5748793Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5749012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5749151Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5749467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5749544Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5749824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5749888Z     self_outputs = self.self(
2025-09-07T08:10:23.5750173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5750279Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5750642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5750778Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5750783Z 
2025-09-07T08:10:23.5750909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5751344Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5751408Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5751626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5751695Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5751979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5752045Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5752321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5752435Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5752710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5752790Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5752794Z 
2025-09-07T08:10:23.5752884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5753238Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5753300Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5753511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5753580Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5753853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5753925Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5754199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5754295Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5754570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5754644Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5754647Z 
2025-09-07T08:10:23.5754738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5755090Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5755188Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5755402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5755502Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5755783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5755856Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5756120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5756189Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5756472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5756575Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5756853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5756981Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5756984Z 
2025-09-07T08:10:23.5757077Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5757475Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5757538Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5757757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5757825Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5758099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5758173Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5758434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5758507Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5758792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5758886Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5759163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5759266Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5759485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5759549Z     return self.act(input)
2025-09-07T08:10:23.5759552Z 
2025-09-07T08:10:23.5759643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5759998Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5760061Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5760274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5760341Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5760625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5760705Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5760962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5761037Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5761319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5761504Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5761781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5761853Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5761857Z 
2025-09-07T08:10:23.5761945Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5762298Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5762361Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5762572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5762639Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5762950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5763020Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5763336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5763400Z     self_outputs = self.self(
2025-09-07T08:10:23.5763695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5763767Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5763770Z 
2025-09-07T08:10:23.5763862Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5764214Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5764278Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5764496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5764563Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5764843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5764911Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5765188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5765248Z     self_outputs = self.self(
2025-09-07T08:10:23.5765523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5765622Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5765968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5766163Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5766167Z 
2025-09-07T08:10:23.5766262Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5766620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5766684Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5766898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5766975Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5767253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5767363Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5767686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5767750Z     self_outputs = self.self(
2025-09-07T08:10:23.5768035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5768109Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5768112Z 
2025-09-07T08:10:23.5768209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5768561Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5768634Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5768849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5768918Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5769238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5769307Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5769628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5769693Z     self_outputs = self.self(
2025-09-07T08:10:23.5769970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5770067Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5770411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5770568Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5770848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5770917Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5770920Z 
2025-09-07T08:10:23.5771013Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5771364Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5771433Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5771648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5771718Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5771995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5772066Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5772343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5772403Z     self_outputs = self.self(
2025-09-07T08:10:23.5772682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5772769Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5773109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5773284Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5773288Z 
2025-09-07T08:10:23.5773361Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5773466Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5773566Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5773637Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5773726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5774081Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5774148Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5774360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5774429Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5774705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5774775Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5775087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5775151Z     self_outputs = self.self(
2025-09-07T08:10:23.5775429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5775567Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5775916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5776055Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5776380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5776528Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5776534Z 
2025-09-07T08:10:23.5776608Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5776706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5777061Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5777127Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5777344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5777412Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5777747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5777815Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5778095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5778159Z     self_outputs = self.self(
2025-09-07T08:10:23.5778438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5778505Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5778508Z 
2025-09-07T08:10:23.5778600Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5778953Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5779014Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5779232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5779298Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5779577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5779731Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5780010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5780075Z     self_outputs = self.self(
2025-09-07T08:10:23.5780350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5780422Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5780425Z 
2025-09-07T08:10:23.5780516Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5780865Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5780936Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5781150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5781259Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5781540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5781644Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5781932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5781997Z     self_outputs = self.self(
2025-09-07T08:10:23.5782280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5782358Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5782361Z 
2025-09-07T08:10:23.5782457Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5782807Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5782874Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5783087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5783157Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5783443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5783511Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5783787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5783855Z     self_outputs = self.self(
2025-09-07T08:10:23.5784131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5784249Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5784609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5784788Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5784983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5785076Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5785079Z 
2025-09-07T08:10:23.5785176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5785527Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5785597Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5785851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5785953Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5786232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5786302Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5786589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5786652Z     self_outputs = self.self(
2025-09-07T08:10:23.5786933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5787037Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5787394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5787570Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5787936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5788019Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5788200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5788286Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5788289Z 
2025-09-07T08:10:23.5788378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5788731Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5788796Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5789011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5789078Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5789361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5789428Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5789703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5789765Z     self_outputs = self.self(
2025-09-07T08:10:23.5790039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5790141Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5790493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5790638Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5790642Z 
2025-09-07T08:10:23.5790732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5791082Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5791145Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5791357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5791422Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5791699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5791799Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5792111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5792171Z     self_outputs = self.self(
2025-09-07T08:10:23.5792447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5792547Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5792895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5793030Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5793033Z 
2025-09-07T08:10:23.5793122Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5793505Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5793569Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5793783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5793882Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5794162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5794232Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5794508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5794611Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5794885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5794967Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5794975Z 
2025-09-07T08:10:23.5795068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5795417Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5795481Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5795691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5795758Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5796034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5796098Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5796377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5796478Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5796758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5796832Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5796835Z 
2025-09-07T08:10:23.5796924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5797272Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5797331Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5797543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5797609Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5797932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5798039Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5798303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5798380Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5798663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5798770Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5799056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5799129Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5799132Z 
2025-09-07T08:10:23.5799222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5799607Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5799673Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5799919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5799995Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5800272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5800344Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5800604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5800669Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5800959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5801058Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5801339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5801442Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5801649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5801712Z     return self.act(input)
2025-09-07T08:10:23.5801716Z 
2025-09-07T08:10:23.5801804Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5802154Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5802218Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5802442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5802509Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5802788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5802865Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5803314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5803387Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5803669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5803781Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5804067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5804261Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5804264Z 
2025-09-07T08:10:23.5804361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5804714Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5804781Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5804994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5805062Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5805343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5805411Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5805738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5805806Z     self_outputs = self.self(
2025-09-07T08:10:23.5806083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5806207Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5806211Z 
2025-09-07T08:10:23.5806300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5806655Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5806715Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5806929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5806995Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5807274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5807352Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5807634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5807701Z     self_outputs = self.self(
2025-09-07T08:10:23.5807975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5808066Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5808412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5808593Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5808598Z 
2025-09-07T08:10:23.5808693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5809041Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5809110Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5809324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5809396Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5809676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5809743Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5810026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5810122Z     self_outputs = self.self(
2025-09-07T08:10:23.5810438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5810509Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5810513Z 
2025-09-07T08:10:23.5810605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5810957Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5811017Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5811234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5811300Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5811586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5811655Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5811967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5812033Z     self_outputs = self.self(
2025-09-07T08:10:23.5812371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5812463Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5812804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5812952Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5813235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5813304Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5813309Z 
2025-09-07T08:10:23.5813406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5813756Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5813826Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5814040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5814106Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5814386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5814453Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5814734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5814797Z     self_outputs = self.self(
2025-09-07T08:10:23.5815075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5815166Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5815507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5815685Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5815688Z 
2025-09-07T08:10:23.5815760Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5815833Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5815901Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5815971Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5816066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5816458Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5816559Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5816771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5816837Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5817115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5817179Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5817459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5817520Z     self_outputs = self.self(
2025-09-07T08:10:23.5817848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5817993Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5818368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5818512Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5818838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5818980Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5818983Z 
2025-09-07T08:10:23.5819054Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5819147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5819501Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5819568Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5819791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5819862Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5820149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5820217Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5820494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5820558Z     self_outputs = self.self(
2025-09-07T08:10:23.5820835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5820902Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5820906Z 
2025-09-07T08:10:23.5820995Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5821345Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5821408Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5821621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5821689Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5821964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5822030Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5822305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5822433Z     self_outputs = self.self(
2025-09-07T08:10:23.5822715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5822784Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5822787Z 
2025-09-07T08:10:23.5822878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5823226Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5823289Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5823502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5823568Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5823848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5823948Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5824234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5824295Z     self_outputs = self.self(
2025-09-07T08:10:23.5824621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5824703Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5824707Z 
2025-09-07T08:10:23.5824793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5825146Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5825208Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5825423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5825493Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5825770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5825841Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5826116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5826181Z     self_outputs = self.self(
2025-09-07T08:10:23.5826462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5826571Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5826929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5827100Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5827291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5827383Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5827387Z 
2025-09-07T08:10:23.5827486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5827838Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5827899Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5828114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5828184Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5828508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5828613Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5828900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5828964Z     self_outputs = self.self(
2025-09-07T08:10:23.5829245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5829355Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5829708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5829836Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5830188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5830275Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5830458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5830574Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5830578Z 
2025-09-07T08:10:23.5830670Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5831021Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5831084Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5831295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5831361Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5831641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5831708Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5831986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5832046Z     self_outputs = self.self(
2025-09-07T08:10:23.5832317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5832419Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5832770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5832915Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5832920Z 
2025-09-07T08:10:23.5833009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5833366Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5833426Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5833640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5833714Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5833990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5834059Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5834336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5834437Z     self_outputs = self.self(
2025-09-07T08:10:23.5834716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5834849Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5835208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5835341Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5835345Z 
2025-09-07T08:10:23.5835435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5835784Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5835850Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5836059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5836160Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5836449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5836547Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5836829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5836930Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5837214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5837292Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5837295Z 
2025-09-07T08:10:23.5837383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5837739Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5837804Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5838018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5838084Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5838360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5838427Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5838700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5838800Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5839074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5839149Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5839154Z 
2025-09-07T08:10:23.5839242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5839590Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5839653Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5839865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5839937Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5840213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5840288Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5840594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5840696Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5840988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5841092Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5841376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5841447Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5841450Z 
2025-09-07T08:10:23.5841544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5841896Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5841961Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5842208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5842276Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5842590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5842666Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5842926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5842996Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5843276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5843373Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5843649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5843755Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5843967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5844029Z     return self.act(input)
2025-09-07T08:10:23.5844033Z 
2025-09-07T08:10:23.5844125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5844472Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5844534Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5844747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5844813Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5845094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5845166Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5845426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5845494Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5845774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5845891Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5846168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5846244Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5846247Z 
2025-09-07T08:10:23.5846334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5846729Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5846822Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5847036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5847105Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5847380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5847449Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5847722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5847784Z     self_outputs = self.self(
2025-09-07T08:10:23.5848062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T08:10:23.5848182Z     query_vectors = self.query(hidden_states)
2025-09-07T08:10:23.5848185Z 
2025-09-07T08:10:23.5848276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5848669Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5848733Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5848945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5849010Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5849289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5849356Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5849637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5849698Z     self_outputs = self.self(
2025-09-07T08:10:23.5849972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5850068Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5850409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5850591Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5850594Z 
2025-09-07T08:10:23.5850685Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5851036Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5851101Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5851312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5851388Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5851665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5851738Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5852014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5852083Z     self_outputs = self.self(
2025-09-07T08:10:23.5852359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T08:10:23.5852429Z     key_vectors = self.key(hidden_states)
2025-09-07T08:10:23.5852466Z 
2025-09-07T08:10:23.5852596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5852950Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5853018Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5853232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5853301Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5853586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5853652Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5853932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5853996Z     self_outputs = self.self(
2025-09-07T08:10:23.5854311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5854404Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5854781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 790, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5854936Z     key = self._chunk(key, window_overlap, getattr(self.config, "onnx_export", False))
2025-09-07T08:10:23.5855218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 719, in _chunk
2025-09-07T08:10:23.5855296Z     hidden_states = hidden_states.view(
2025-09-07T08:10:23.5855299Z 
2025-09-07T08:10:23.5855389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5855748Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5855817Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5856032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5856104Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5856380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5856453Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5856731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5856792Z     self_outputs = self.self(
2025-09-07T08:10:23.5857071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T08:10:23.5857162Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5857516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5857762Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T08:10:23.5857766Z 
2025-09-07T08:10:23.5857842Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5857911Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5857979Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5858053Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5858142Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5858497Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5858598Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5858815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5858919Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5859199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5859271Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5859547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5859608Z     self_outputs = self.self(
2025-09-07T08:10:23.5859890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T08:10:23.5859990Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T08:10:23.5860366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T08:10:23.5860511Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T08:10:23.5860875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T08:10:23.5861014Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T08:10:23.5861017Z 
2025-09-07T08:10:23.5861088Z cudagraph partition due to non gpu ops
2025-09-07T08:10:23.5861183Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5861536Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5861604Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5861819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5861899Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5862179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5862247Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5862531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5862591Z     self_outputs = self.self(
2025-09-07T08:10:23.5862870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T08:10:23.5862936Z     attn_scores += diagonal_mask
2025-09-07T08:10:23.5862939Z 
2025-09-07T08:10:23.5863028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5863382Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5863447Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5863664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5863733Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5864016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5864083Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5864360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5864426Z     self_outputs = self.self(
2025-09-07T08:10:23.5864698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T08:10:23.5864810Z     attn_probs = nn.functional.softmax(
2025-09-07T08:10:23.5864846Z 
2025-09-07T08:10:23.5864937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5865287Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5865355Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5865566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5865641Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5865918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5865988Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5866263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5866327Z     self_outputs = self.self(
2025-09-07T08:10:23.5866641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T08:10:23.5866719Z     value_vectors = self.value(hidden_states)
2025-09-07T08:10:23.5866750Z 
2025-09-07T08:10:23.5866848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5867198Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5867263Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5867476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5867542Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5867825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5867895Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5868180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5868242Z     self_outputs = self.self(
2025-09-07T08:10:23.5868516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5868632Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5868986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5869156Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T08:10:23.5869345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5869443Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5869446Z 
2025-09-07T08:10:23.5869539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5869896Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5869968Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5870190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5870274Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5870560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5870632Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5870916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5871061Z     self_outputs = self.self(
2025-09-07T08:10:23.5871352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5871469Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5871830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5871961Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T08:10:23.5872282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T08:10:23.5872368Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T08:10:23.5872556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T08:10:23.5872691Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T08:10:23.5872694Z 
2025-09-07T08:10:23.5872793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5873182Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5873250Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5873471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5873548Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5873831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5873904Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5874185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5874251Z     self_outputs = self.self(
2025-09-07T08:10:23.5874537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5874644Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5875000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5875147Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5875150Z 
2025-09-07T08:10:23.5875247Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5875598Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5875662Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5875884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5875954Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5876238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5876304Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5876587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:10:23.5876652Z     self_outputs = self.self(
2025-09-07T08:10:23.5876929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T08:10:23.5877032Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T08:10:23.5877421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T08:10:23.5877596Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T08:10:23.5877600Z 
2025-09-07T08:10:23.5877692Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5878042Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5878110Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5878324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5878396Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5878675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5878784Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5879067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5879201Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5879486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5879563Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5879566Z 
2025-09-07T08:10:23.5879664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5880011Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5880079Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5880295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5880364Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5880646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:10:23.5880715Z     self_attn_outputs = self.attention(
2025-09-07T08:10:23.5880996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T08:10:23.5881092Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:10:23.5881369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T08:10:23.5881447Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5881450Z 
2025-09-07T08:10:23.5881543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5881903Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5881966Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5882185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5882255Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5882532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5882609Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5882871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5882948Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5883235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5883403Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5883685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T08:10:23.5883760Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5883763Z 
2025-09-07T08:10:23.5883861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5884213Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5884280Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5884492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5884561Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5884880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5884954Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5885254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5885324Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5885611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T08:10:23.5885710Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T08:10:23.5885992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T08:10:23.5886102Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:10:23.5886311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:10:23.5886384Z     return self.act(input)
2025-09-07T08:10:23.5886387Z 
2025-09-07T08:10:23.5886477Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:10:23.5886828Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T08:10:23.5886899Z     layer_outputs = layer_module(
2025-09-07T08:10:23.5887113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:10:23.5887187Z     return super().__call__(*args, **kwargs)
2025-09-07T08:10:23.5887466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T08:10:23.5887548Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:10:23.5887811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:10:23.5887882Z     return forward_fn(*input_tensors)
2025-09-07T08:10:23.5888169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T08:10:23.5888283Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T08:10:23.5888566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T08:10:23.5888639Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:10:23.5888642Z 
2025-09-07T08:11:27.7024171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:27.7024831Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703
2025-09-07T08:11:27.7025865Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T08:11:27.7026476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward
2025-09-07T08:11:27.7026922Z     x = self.dense(features)
2025-09-07T08:11:27.7027037Z 
2025-09-07T08:11:27.7027152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:27.7027698Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703
2025-09-07T08:11:27.7028222Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T08:11:27.7028679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward
2025-09-07T08:11:27.7029091Z     x = self.decoder(x)
2025-09-07T08:11:27.7029192Z 
2025-09-07T08:11:30.2132580Z pass
2025-09-07T08:11:30.2132989Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:11:34.5957358Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:11:34.5958242Z   import pynvml  # type: ignore[import]
2025-09-07T08:11:36.6133919Z 
2025-09-07T08:11:38.0337264Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:11:38.0338752Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:11:38.8074858Z 
2025-09-07T08:11:38.8075350Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:11:38.8126092Z cpu  eval  hf_Reformer                        
2025-09-07T08:11:39.3268720Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:11:40.8352800Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:11:42.3825129Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:11:48.2491992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:48.2492555Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2095, in forward
2025-09-07T08:11:48.2493088Z     least_common_mult_chunk_length = _get_least_common_mult_chunk_len(self.config)
2025-09-07T08:11:48.2493697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 194, in _get_least_common_mult_chunk_len
2025-09-07T08:11:48.2494279Z     return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length)
2025-09-07T08:11:48.2494494Z 
2025-09-07T08:11:48.2494611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:48.2495064Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2099, in forward
2025-09-07T08:11:48.2495518Z     input_shape[-1] % least_common_mult_chunk_length != 0
2025-09-07T08:11:48.2495680Z 
2025-09-07T08:11:48.6720067Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:11:48.6720753Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:11:48.6721604Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] or:
2025-09-07T08:11:48.6722203Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:11:48.6722783Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] to include these operations in the captured graph.
2025-09-07T08:11:48.6723259Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:11:48.6723746Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break: from user code at:
2025-09-07T08:11:48.6724613Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2132, in torch_dynamo_resume_in_forward_at_2104
2025-09-07T08:11:48.6725512Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     embedding_output = self.embeddings(
2025-09-07T08:11:48.6726373Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 369, in forward
2025-09-07T08:11:48.6727241Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     position_embeddings = self.position_embeddings(position_ids)
2025-09-07T08:11:48.6728039Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T08:11:48.6728791Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     max_position_id = position_ids.max().item()
2025-09-07T08:11:48.6729258Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:11:48.6729637Z W0907 08:11:48.671357 56129 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:11:48.8052588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:48.8053159Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 351, in forward
2025-09-07T08:11:48.8053599Z     position_ids = torch.arange(
2025-09-07T08:11:48.8053730Z 
2025-09-07T08:11:48.8053837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:48.8054288Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 357, in forward
2025-09-07T08:11:48.8054722Z     inputs_embeds = self.word_embeddings(input_ids)
2025-09-07T08:11:48.8054873Z 
2025-09-07T08:11:49.5570884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:49.5571441Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T08:11:49.5571946Z     max_position_id = position_ids.max().item()
2025-09-07T08:11:49.5572091Z 
2025-09-07T08:11:51.2869060Z cudagraph partition due to non gpu ops
2025-09-07T08:11:51.2869347Z cudagraph partition due to non gpu ops
2025-09-07T08:11:51.2869629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:51.2870200Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 297, in torch_dynamo_resume_in_forward_at_293
2025-09-07T08:11:51.2870712Z     position_encodings = torch.cat(
2025-09-07T08:11:51.2870849Z 
2025-09-07T08:11:51.2870965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:51.2871484Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 303, in torch_dynamo_resume_in_forward_at_293
2025-09-07T08:11:51.2871976Z     position_encodings = torch.cat(
2025-09-07T08:11:51.2872462Z 
2025-09-07T08:11:52.2203406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:52.2204403Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 370, in torch_dynamo_resume_in_forward_at_369
2025-09-07T08:11:52.2204960Z     embeddings = embeddings + position_embeddings
2025-09-07T08:11:52.2205126Z 
2025-09-07T08:11:55.3797591Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3798227Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3798763Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3799227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3799650Z     layer_outputs = layer(
2025-09-07T08:11:55.3800394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3800848Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3801286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3801826Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3802305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3802790Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3803556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 749, in _hash_vectors
2025-09-07T08:11:55.3804108Z     random_rotations = torch.randn(rotations_shape, device=vectors.device, dtype=vectors.dtype)
2025-09-07T08:11:55.3804362Z 
2025-09-07T08:11:55.3804472Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3805003Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3805510Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3805938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3806348Z     layer_outputs = layer(
2025-09-07T08:11:55.3806731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3807147Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3807538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3807970Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3808431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3808973Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3809593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 807, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3810150Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2025-09-07T08:11:55.3810328Z 
2025-09-07T08:11:55.3810445Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3810945Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3811438Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3811940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3812436Z     layer_outputs = layer(
2025-09-07T08:11:55.3812821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3813242Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3813629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3814053Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3814471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3814990Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3815654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 800, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3816216Z     torch.arange(sorted_bucket_idx.shape[-1], device=buckets.device)
2025-09-07T08:11:55.3816412Z 
2025-09-07T08:11:55.3816511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3817068Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3817567Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3818044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3818440Z     layer_outputs = layer(
2025-09-07T08:11:55.3818817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3819229Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3819626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3820043Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3820459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3820941Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3821435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 749, in _hash_vectors
2025-09-07T08:11:55.3821975Z     random_rotations = torch.randn(rotations_shape, device=vectors.device, dtype=vectors.dtype)
2025-09-07T08:11:55.3822219Z 
2025-09-07T08:11:55.3822321Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3822819Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3823320Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3823732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3824134Z     layer_outputs = layer(
2025-09-07T08:11:55.3824509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3824909Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3825307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3825738Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3826179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3826763Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3827406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 807, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3827961Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2025-09-07T08:11:55.3828149Z 
2025-09-07T08:11:55.3828249Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3828761Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3829259Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3829682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3830082Z     layer_outputs = layer(
2025-09-07T08:11:55.3830508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3830919Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3831355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3831790Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3832220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3832777Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3833414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 800, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3833974Z     torch.arange(sorted_bucket_idx.shape[-1], device=buckets.device)
2025-09-07T08:11:55.3834172Z 
2025-09-07T08:11:55.3834275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3834784Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3835286Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3835701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3836101Z     layer_outputs = layer(
2025-09-07T08:11:55.3836476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3836881Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3837265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3837686Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3838101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3838570Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3839064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 749, in _hash_vectors
2025-09-07T08:11:55.3839607Z     random_rotations = torch.randn(rotations_shape, device=vectors.device, dtype=vectors.dtype)
2025-09-07T08:11:55.3839855Z 
2025-09-07T08:11:55.3839963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3840480Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3840967Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3841430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3841871Z     layer_outputs = layer(
2025-09-07T08:11:55.3842257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3842658Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3843048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3843468Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3843887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3844409Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3845061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 807, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3845613Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2025-09-07T08:11:55.3845801Z 
2025-09-07T08:11:55.3845933Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3846442Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3846937Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3847360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3847754Z     layer_outputs = layer(
2025-09-07T08:11:55.3857974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3858471Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3858909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3859357Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3859792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3860325Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3860952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 800, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3861506Z     torch.arange(sorted_bucket_idx.shape[-1], device=buckets.device)
2025-09-07T08:11:55.3861702Z 
2025-09-07T08:11:55.3861811Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3862328Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3862835Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3863277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3863678Z     layer_outputs = layer(
2025-09-07T08:11:55.3864060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3864468Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3864869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3865296Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3865704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1198, in forward
2025-09-07T08:11:55.3866242Z     query_vectors = self.query(hidden_states)
2025-09-07T08:11:55.3866384Z 
2025-09-07T08:11:55.3866483Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3866990Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3867501Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3867915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3868314Z     layer_outputs = layer(
2025-09-07T08:11:55.3868690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3869094Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3869525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3869942Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3870391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1199, in forward
2025-09-07T08:11:55.3870805Z     key_vectors = self.key(hidden_states)
2025-09-07T08:11:55.3870936Z 
2025-09-07T08:11:55.3871022Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3871221Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3871445Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3871945Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3872433Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3872841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3873236Z     layer_outputs = layer(
2025-09-07T08:11:55.3873612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3874012Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3874403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3874819Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3875221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T08:11:55.3875734Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.3876267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.3876771Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.3876977Z 
2025-09-07T08:11:55.3877061Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3877267Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3877491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3877992Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3878486Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3878889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3879280Z     layer_outputs = layer(
2025-09-07T08:11:55.3879664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3881544Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3881944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3882377Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3882789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T08:11:55.3883298Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.3883837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.3884260Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.3884384Z 
2025-09-07T08:11:55.3884481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3885023Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3885520Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3885968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3886366Z     layer_outputs = layer(
2025-09-07T08:11:55.3886733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3887135Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3887524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3887938Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3888347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T08:11:55.3888828Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.3889046Z 
2025-09-07T08:11:55.3889141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3889652Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3890149Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3890557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3890954Z     layer_outputs = layer(
2025-09-07T08:11:55.3891332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3891739Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3892129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3892539Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3892947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T08:11:55.3893420Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.3893619Z 
2025-09-07T08:11:55.3893721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3894216Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3894693Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3895114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3895588Z     layer_outputs = layer(
2025-09-07T08:11:55.3896008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3896411Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3896797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3897211Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3897669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1200, in forward
2025-09-07T08:11:55.3898089Z     value_vectors = self.value(hidden_states)
2025-09-07T08:11:55.3898223Z 
2025-09-07T08:11:55.3898305Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3898500Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3898721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3899275Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3899784Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3900242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3900653Z     layer_outputs = layer(
2025-09-07T08:11:55.3901039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3901442Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3901841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3902260Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3902674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1266, in forward
2025-09-07T08:11:55.3903412Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.3903965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.3904466Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.3904673Z 
2025-09-07T08:11:55.3904755Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3904960Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3905185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3905687Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3906181Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3906597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3906996Z     layer_outputs = layer(
2025-09-07T08:11:55.3907370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3907771Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3908161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3908573Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3908992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T08:11:55.3909435Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.3909604Z 
2025-09-07T08:11:55.3909707Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3910286Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3910823Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3911234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3911633Z     layer_outputs = layer(
2025-09-07T08:11:55.3912005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3912405Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3912804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3913227Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3913646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T08:11:55.3914156Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.3914329Z 
2025-09-07T08:11:55.3914429Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3914999Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3915495Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3915911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3916313Z     layer_outputs = layer(
2025-09-07T08:11:55.3916680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3917080Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3917475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3917917Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3918326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1324, in forward
2025-09-07T08:11:55.3918858Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T08:11:55.3919441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T08:11:55.3919959Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T08:11:55.3920152Z 
2025-09-07T08:11:55.3920255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3920756Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3921240Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3921652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3922053Z     layer_outputs = layer(
2025-09-07T08:11:55.3922427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3922830Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3923219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T08:11:55.3923681Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T08:11:55.3924152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T08:11:55.3924647Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.3924784Z 
2025-09-07T08:11:55.3925027Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3925533Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3926024Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3926438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3926837Z     layer_outputs = layer(
2025-09-07T08:11:55.3927210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.3927666Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.3928107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.3928555Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.3928960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.3929391Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.3929803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.3930242Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.3930652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T08:11:55.3931062Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.3931192Z 
2025-09-07T08:11:55.3931299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3931797Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3932288Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3932704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3933103Z     layer_outputs = layer(
2025-09-07T08:11:55.3933477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.3933920Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.3934367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.3934777Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.3935175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.3935573Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.3935981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.3936409Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.3936817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T08:11:55.3937230Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T08:11:55.3937366Z 
2025-09-07T08:11:55.3937467Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3938026Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3938522Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3938989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3939443Z     layer_outputs = layer(
2025-09-07T08:11:55.3939825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.3940272Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.3940722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.3941133Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.3941522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.3941910Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.3942325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T08:11:55.3942790Z     return self.output(hidden_states)
2025-09-07T08:11:55.3943187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T08:11:55.3943634Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.3943769Z 
2025-09-07T08:11:55.3943868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3944369Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3944863Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3945281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3945679Z     layer_outputs = layer(
2025-09-07T08:11:55.3946054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3946463Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3946857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3947280Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3947700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 540, in forward
2025-09-07T08:11:55.3948119Z     query_key_vectors = self.query_key(hidden_states)
2025-09-07T08:11:55.3948271Z 
2025-09-07T08:11:55.3948365Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3948862Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3949353Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3949765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3950157Z     layer_outputs = layer(
2025-09-07T08:11:55.3950530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3950931Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3951320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3951730Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3952145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3952622Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3953121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 749, in _hash_vectors
2025-09-07T08:11:55.3953734Z     random_rotations = torch.randn(rotations_shape, device=vectors.device, dtype=vectors.dtype)
2025-09-07T08:11:55.3953981Z 
2025-09-07T08:11:55.3954090Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3954587Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3955078Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3955492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3955890Z     layer_outputs = layer(
2025-09-07T08:11:55.3956264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3956669Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3957093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3957523Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3957969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3958435Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3958926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 751, in _hash_vectors
2025-09-07T08:11:55.3959412Z     rotated_vectors = torch.einsum("bmtd,mdhr->bmhtr", vectors, random_rotations)
2025-09-07T08:11:55.3959616Z 
2025-09-07T08:11:55.3959714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3960203Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3960686Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3961090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3961479Z     layer_outputs = layer(
2025-09-07T08:11:55.3961844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3962236Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3962613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3963018Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3963416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3963876Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3964351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 754, in _hash_vectors
2025-09-07T08:11:55.3964826Z     rotated_vectors = torch.cat([rotated_vectors, -rotated_vectors], dim=-1)
2025-09-07T08:11:55.3965025Z 
2025-09-07T08:11:55.3965120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3965607Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3966092Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3966491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3966873Z     layer_outputs = layer(
2025-09-07T08:11:55.3967285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3967731Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3968119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3968528Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3968928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.3969387Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.3969854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 755, in _hash_vectors
2025-09-07T08:11:55.3970289Z     buckets = torch.argmax(rotated_vectors, dim=-1)
2025-09-07T08:11:55.3970434Z 
2025-09-07T08:11:55.3970533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3971069Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3971557Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3971999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3972397Z     layer_outputs = layer(
2025-09-07T08:11:55.3972765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3973158Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3973543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3973958Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3974364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3974876Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3975484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3976005Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T08:11:55.3976447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 182, in _stable_argsort
2025-09-07T08:11:55.3976938Z     scaled_vector = vector.shape[dim] * vector + (scale_offset % vector.shape[dim])
2025-09-07T08:11:55.3977142Z 
2025-09-07T08:11:55.3977222Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3977446Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3978009Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3978494Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3978910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3979301Z     layer_outputs = layer(
2025-09-07T08:11:55.3979676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3980071Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3980461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3980874Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3981287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.3981900Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.3982511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.3983028Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T08:11:55.3983468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 183, in _stable_argsort
2025-09-07T08:11:55.3983907Z     return torch.argsort(scaled_vector, dim=dim)
2025-09-07T08:11:55.3984053Z 
2025-09-07T08:11:55.3984125Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3984320Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3984532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3985069Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3985555Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3986006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3986401Z     layer_outputs = layer(
2025-09-07T08:11:55.3986788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3987204Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3987589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3987996Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3988424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.3988846Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.3989268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T08:11:55.3989808Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.3990358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.3990845Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.3991045Z 
2025-09-07T08:11:55.3991119Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3991314Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3991526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3992025Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3992509Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3992925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.3993318Z     layer_outputs = layer(
2025-09-07T08:11:55.3993682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.3994092Z     attn_outputs = self.attention(
2025-09-07T08:11:55.3994475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.3994897Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.3995300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.3995872Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.3996316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T08:11:55.3996877Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.3997417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.3997852Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.3997984Z 
2025-09-07T08:11:55.3998054Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.3998304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.3998805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.3999297Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.3999819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4000217Z     layer_outputs = layer(
2025-09-07T08:11:55.4000630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4001040Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4001426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4001845Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4002254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 541, in forward
2025-09-07T08:11:55.4002669Z     value_vectors = self.value(hidden_states)
2025-09-07T08:11:55.4002808Z 
2025-09-07T08:11:55.4002913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4003564Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4004063Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4004473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4004867Z     layer_outputs = layer(
2025-09-07T08:11:55.4005240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4005637Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4006030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4006457Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4006869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4007296Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4007712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T08:11:55.4008221Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4008747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4009237Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4009440Z 
2025-09-07T08:11:55.4009522Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4009740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4010325Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4010877Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4011303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4011707Z     layer_outputs = layer(
2025-09-07T08:11:55.4012081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4012488Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4012883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4013302Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4013759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4014198Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4014673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T08:11:55.4015189Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4015720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4016136Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4016266Z 
2025-09-07T08:11:55.4016365Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4016869Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4017359Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4017859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4018253Z     layer_outputs = layer(
2025-09-07T08:11:55.4018630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4019031Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4019423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4019848Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4020255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 595, in forward
2025-09-07T08:11:55.4020795Z     query_key_vectors = self._gather_by_expansion(query_key_vectors, sorted_bucket_idx_per_hash, num_hashes)
2025-09-07T08:11:55.4021379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1099, in _gather_by_expansion
2025-09-07T08:11:55.4021846Z     return torch.gather(vectors, 2, expanded_idxs)
2025-09-07T08:11:55.4021991Z 
2025-09-07T08:11:55.4022094Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4022587Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4023075Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4023484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4023880Z     layer_outputs = layer(
2025-09-07T08:11:55.4024251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4024742Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4025131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4025548Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4025961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4026390Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4026806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 849, in _attend
2025-09-07T08:11:55.4027281Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4027492Z 
2025-09-07T08:11:55.4027587Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4028130Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4028627Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4029071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4029474Z     layer_outputs = layer(
2025-09-07T08:11:55.4029845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4030244Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4030623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4031043Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4031450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4031883Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4032300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 845, in _attend
2025-09-07T08:11:55.4032816Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4033356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4033849Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4034066Z 
2025-09-07T08:11:55.4034142Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4034371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4034864Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4035359Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4035775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4036175Z     layer_outputs = layer(
2025-09-07T08:11:55.4036552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4036951Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4037340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4037766Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4038183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4038613Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4039127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T08:11:55.4039569Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4039743Z 
2025-09-07T08:11:55.4039841Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4040339Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4040826Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4041231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4041628Z     layer_outputs = layer(
2025-09-07T08:11:55.4042002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4042406Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4042833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4043249Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4043700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4044130Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4044555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T08:11:55.4044986Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4045155Z 
2025-09-07T08:11:55.4045231Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4045458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4045955Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4046444Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4046849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4047244Z     layer_outputs = layer(
2025-09-07T08:11:55.4047611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4048015Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4048404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4048812Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4049221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 692, in forward
2025-09-07T08:11:55.4049762Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T08:11:55.4050347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T08:11:55.4050864Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T08:11:55.4051058Z 
2025-09-07T08:11:55.4051154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4051653Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4052142Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4052545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4053009Z     layer_outputs = layer(
2025-09-07T08:11:55.4053375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4053776Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4054167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T08:11:55.4054636Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T08:11:55.4055103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T08:11:55.4055510Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4055652Z 
2025-09-07T08:11:55.4055747Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4056239Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4056763Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4057167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4057590Z     layer_outputs = layer(
2025-09-07T08:11:55.4058002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4058451Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4058898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4059301Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4059684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4060077Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4060488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4060916Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4061316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T08:11:55.4061726Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4061866Z 
2025-09-07T08:11:55.4061965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4062465Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4062955Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4063355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4063760Z     layer_outputs = layer(
2025-09-07T08:11:55.4064133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4064582Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4065030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4065427Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4065811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4066205Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4066619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4067094Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4067523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T08:11:55.4067936Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T08:11:55.4068075Z 
2025-09-07T08:11:55.4068169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4068664Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4069149Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4069548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4069945Z     layer_outputs = layer(
2025-09-07T08:11:55.4070315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4070796Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4071239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4071695Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4072085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4072479Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4072890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T08:11:55.4073307Z     return self.output(hidden_states)
2025-09-07T08:11:55.4073700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T08:11:55.4074112Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4074245Z 
2025-09-07T08:11:55.4074345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4074840Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4075321Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4075743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4076135Z     layer_outputs = layer(
2025-09-07T08:11:55.4076514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4076919Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4077301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4077722Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4078137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1198, in forward
2025-09-07T08:11:55.4078547Z     query_vectors = self.query(hidden_states)
2025-09-07T08:11:55.4078676Z 
2025-09-07T08:11:55.4078778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4079264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4079748Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4080152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4080549Z     layer_outputs = layer(
2025-09-07T08:11:55.4080920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4081411Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4081823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4082238Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4082659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1199, in forward
2025-09-07T08:11:55.4083062Z     key_vectors = self.key(hidden_states)
2025-09-07T08:11:55.4083201Z 
2025-09-07T08:11:55.4083277Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4083480Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4083700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4084197Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4084675Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4085138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4085541Z     layer_outputs = layer(
2025-09-07T08:11:55.4085955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4086362Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4086747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4087168Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4087578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T08:11:55.4088097Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4088628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4089133Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4089347Z 
2025-09-07T08:11:55.4089420Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4089625Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4089851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4090345Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4090836Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4091246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4091651Z     layer_outputs = layer(
2025-09-07T08:11:55.4092022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4092415Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4092797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4093210Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4093621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T08:11:55.4094125Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4094640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4095062Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4095241Z 
2025-09-07T08:11:55.4095368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4095874Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4096359Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4096762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4097162Z     layer_outputs = layer(
2025-09-07T08:11:55.4097553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4098019Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4098414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4098831Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4099289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T08:11:55.4099771Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4099975Z 
2025-09-07T08:11:55.4100111Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4100608Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4101086Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4101502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4101903Z     layer_outputs = layer(
2025-09-07T08:11:55.4102273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4102673Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4103228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4103649Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4104064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T08:11:55.4104537Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4104731Z 
2025-09-07T08:11:55.4104824Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4105338Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4105827Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4106247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4106650Z     layer_outputs = layer(
2025-09-07T08:11:55.4107021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4107419Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4107805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4108223Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4108629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1200, in forward
2025-09-07T08:11:55.4109035Z     value_vectors = self.value(hidden_states)
2025-09-07T08:11:55.4109174Z 
2025-09-07T08:11:55.4109248Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4109548Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4109822Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4110319Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4110807Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4111213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4111617Z     layer_outputs = layer(
2025-09-07T08:11:55.4111985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4112377Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4112768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4113187Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4113645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1266, in forward
2025-09-07T08:11:55.4114228Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4114771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4115265Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4115478Z 
2025-09-07T08:11:55.4115555Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4115759Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4115983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4116482Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4116976Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4117383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4117785Z     layer_outputs = layer(
2025-09-07T08:11:55.4118156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4118549Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4118938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4119349Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4119759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T08:11:55.4120199Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4120378Z 
2025-09-07T08:11:55.4120476Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4120978Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4121460Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4121866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4122258Z     layer_outputs = layer(
2025-09-07T08:11:55.4122632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4123028Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4123415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4123876Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4124315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T08:11:55.4124753Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4124926Z 
2025-09-07T08:11:55.4125020Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4125517Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4126005Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4126424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4126817Z     layer_outputs = layer(
2025-09-07T08:11:55.4127186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4127641Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4128030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4128483Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4128894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1324, in forward
2025-09-07T08:11:55.4129433Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T08:11:55.4130014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T08:11:55.4130527Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T08:11:55.4130723Z 
2025-09-07T08:11:55.4130817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4131319Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4131804Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4132211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4132607Z     layer_outputs = layer(
2025-09-07T08:11:55.4132970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4133366Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4133754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T08:11:55.4134223Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T08:11:55.4134694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T08:11:55.4135100Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4135247Z 
2025-09-07T08:11:55.4135342Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4135838Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4136325Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4136730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4137123Z     layer_outputs = layer(
2025-09-07T08:11:55.4137494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4138084Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4138576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4138975Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4139372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4139767Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4140180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4140612Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4141010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T08:11:55.4141420Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4141562Z 
2025-09-07T08:11:55.4141698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4142197Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4142735Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4143138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4143534Z     layer_outputs = layer(
2025-09-07T08:11:55.4143909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4144357Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4144801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4145207Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4145598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4145990Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4146404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4146827Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4147231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T08:11:55.4147642Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T08:11:55.4147780Z 
2025-09-07T08:11:55.4147875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4148377Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4148867Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4149270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4149670Z     layer_outputs = layer(
2025-09-07T08:11:55.4150044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4150494Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4150926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4151341Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4151726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4152161Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4152610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T08:11:55.4153018Z     return self.output(hidden_states)
2025-09-07T08:11:55.4153422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T08:11:55.4153820Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4153951Z 
2025-09-07T08:11:55.4154044Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4154538Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4155011Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4155412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4155824Z     layer_outputs = layer(
2025-09-07T08:11:55.4156221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4156615Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4157053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4157480Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4157896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 540, in forward
2025-09-07T08:11:55.4158316Z     query_key_vectors = self.query_key(hidden_states)
2025-09-07T08:11:55.4158463Z 
2025-09-07T08:11:55.4158562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4159056Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4159539Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4159940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4160334Z     layer_outputs = layer(
2025-09-07T08:11:55.4160696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4161089Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4161473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4161880Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4162288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4162760Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4163257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 749, in _hash_vectors
2025-09-07T08:11:55.4163792Z     random_rotations = torch.randn(rotations_shape, device=vectors.device, dtype=vectors.dtype)
2025-09-07T08:11:55.4164042Z 
2025-09-07T08:11:55.4164135Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4164624Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4172957Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4173416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4173846Z     layer_outputs = layer(
2025-09-07T08:11:55.4174326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4174776Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4175174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4175599Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4176023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4176508Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4176993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 751, in _hash_vectors
2025-09-07T08:11:55.4177488Z     rotated_vectors = torch.einsum("bmtd,mdhr->bmhtr", vectors, random_rotations)
2025-09-07T08:11:55.4177776Z 
2025-09-07T08:11:55.4177882Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4178446Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4178948Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4179417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4179822Z     layer_outputs = layer(
2025-09-07T08:11:55.4180199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4180602Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4180990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4181401Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4181813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4182277Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4182758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 754, in _hash_vectors
2025-09-07T08:11:55.4183237Z     rotated_vectors = torch.cat([rotated_vectors, -rotated_vectors], dim=-1)
2025-09-07T08:11:55.4183434Z 
2025-09-07T08:11:55.4183534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4184042Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4184530Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4184935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4185334Z     layer_outputs = layer(
2025-09-07T08:11:55.4185701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4186098Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4186482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4186891Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4187298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4187767Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4188251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 755, in _hash_vectors
2025-09-07T08:11:55.4188777Z     buckets = torch.argmax(rotated_vectors, dim=-1)
2025-09-07T08:11:55.4188986Z 
2025-09-07T08:11:55.4189086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4189590Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4190081Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4190494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4190890Z     layer_outputs = layer(
2025-09-07T08:11:55.4191265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4191664Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4192053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4192511Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4192925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.4193483Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.4194096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.4194622Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T08:11:55.4195070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 182, in _stable_argsort
2025-09-07T08:11:55.4195573Z     scaled_vector = vector.shape[dim] * vector + (scale_offset % vector.shape[dim])
2025-09-07T08:11:55.4195782Z 
2025-09-07T08:11:55.4195867Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4196095Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4196594Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4197086Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4197501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4197897Z     layer_outputs = layer(
2025-09-07T08:11:55.4198265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4198675Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4199066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4199482Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4199895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.4200413Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.4201016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.4201526Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T08:11:55.4201963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 183, in _stable_argsort
2025-09-07T08:11:55.4202396Z     return torch.argsort(scaled_vector, dim=dim)
2025-09-07T08:11:55.4202535Z 
2025-09-07T08:11:55.4202610Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4202848Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4203280Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4203774Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4204260Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4204664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4205057Z     layer_outputs = layer(
2025-09-07T08:11:55.4205425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4205821Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4206196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4206609Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4207104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4207534Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4207999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T08:11:55.4208544Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4209096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4209588Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4209789Z 
2025-09-07T08:11:55.4209872Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4210075Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4210298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4210803Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4211296Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4211716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4212113Z     layer_outputs = layer(
2025-09-07T08:11:55.4212480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4212878Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4213267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4213682Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4214094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4214521Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4214947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T08:11:55.4215469Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4216011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4216421Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4216545Z 
2025-09-07T08:11:55.4216617Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4216834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4217387Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4217983Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4218392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4218782Z     layer_outputs = layer(
2025-09-07T08:11:55.4219145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4219534Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4219914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4220321Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4220723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 541, in forward
2025-09-07T08:11:55.4221164Z     value_vectors = self.value(hidden_states)
2025-09-07T08:11:55.4221295Z 
2025-09-07T08:11:55.4221390Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4221987Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4222469Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4222868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4223257Z     layer_outputs = layer(
2025-09-07T08:11:55.4223617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4224010Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4224386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4224790Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4225189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4225603Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4226011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T08:11:55.4226511Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4227028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4227506Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4227706Z 
2025-09-07T08:11:55.4227786Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4228003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4228500Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4228986Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4229399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4229794Z     layer_outputs = layer(
2025-09-07T08:11:55.4230157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4230554Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4230941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4231438Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4231849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4232269Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4232686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T08:11:55.4233192Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4233717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4234139Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4234265Z 
2025-09-07T08:11:55.4234364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4234895Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4235389Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4235828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4236222Z     layer_outputs = layer(
2025-09-07T08:11:55.4236584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4236976Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4237352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4237758Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4238155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 595, in forward
2025-09-07T08:11:55.4238681Z     query_key_vectors = self._gather_by_expansion(query_key_vectors, sorted_bucket_idx_per_hash, num_hashes)
2025-09-07T08:11:55.4239247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1099, in _gather_by_expansion
2025-09-07T08:11:55.4239703Z     return torch.gather(vectors, 2, expanded_idxs)
2025-09-07T08:11:55.4239844Z 
2025-09-07T08:11:55.4239942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4240434Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4240911Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4241310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4241709Z     layer_outputs = layer(
2025-09-07T08:11:55.4242078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4242473Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4242848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4243254Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4243655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4244075Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4244487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 849, in _attend
2025-09-07T08:11:55.4244953Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4245255Z 
2025-09-07T08:11:55.4245351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4245879Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4246365Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4246771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4247158Z     layer_outputs = layer(
2025-09-07T08:11:55.4247530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4247925Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4248308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4248714Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4249166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4249589Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4250350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 845, in _attend
2025-09-07T08:11:55.4250875Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4251405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4251891Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4252100Z 
2025-09-07T08:11:55.4252174Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4252272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4252616Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4252696Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4252968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4253027Z     layer_outputs = layer(
2025-09-07T08:11:55.4253295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4253360Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4253623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4253704Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4253968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4254065Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4254333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T08:11:55.4254444Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4254451Z 
2025-09-07T08:11:55.4254546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4254885Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4254970Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4255235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4255345Z     layer_outputs = layer(
2025-09-07T08:11:55.4255650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4255718Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4255996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4256080Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4256351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4256444Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4256715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T08:11:55.4256818Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4256823Z 
2025-09-07T08:11:55.4256895Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4257023Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4257365Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4257478Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4257797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4257859Z     layer_outputs = layer(
2025-09-07T08:11:55.4258129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4258196Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4258468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4258552Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4258822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 692, in forward
2025-09-07T08:11:55.4259025Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T08:11:55.4259336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T08:11:55.4259473Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T08:11:55.4259476Z 
2025-09-07T08:11:55.4259566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4259905Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4259986Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4260260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4260322Z     layer_outputs = layer(
2025-09-07T08:11:55.4260592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4260661Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4260927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T08:11:55.4261063Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T08:11:55.4261326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T08:11:55.4261403Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4261410Z 
2025-09-07T08:11:55.4261537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4261917Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4261999Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4262266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4262329Z     layer_outputs = layer(
2025-09-07T08:11:55.4262593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1601, in forward
2025-09-07T08:11:55.4262675Z     attn_output = prev_attn_output + attn_output
2025-09-07T08:11:55.4262678Z 
2025-09-07T08:11:55.4262769Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4263103Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4263219Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4263487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4263581Z     layer_outputs = layer(
2025-09-07T08:11:55.4263851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4263970Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4264242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4264313Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4264585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4264654Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4264950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4265033Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4265302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T08:11:55.4265382Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4265385Z 
2025-09-07T08:11:55.4265478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4265822Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4265903Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4266168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4266238Z     layer_outputs = layer(
2025-09-07T08:11:55.4266505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4266626Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4266899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4266966Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4267224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4267291Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4267578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4267692Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4267998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T08:11:55.4268074Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T08:11:55.4268077Z 
2025-09-07T08:11:55.4268170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4268518Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4268597Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4268871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4268933Z     layer_outputs = layer(
2025-09-07T08:11:55.4269196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4269343Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4269613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4269684Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4269985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4270054Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4270338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T08:11:55.4270405Z     return self.output(hidden_states)
2025-09-07T08:11:55.4270676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T08:11:55.4270749Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4270754Z 
2025-09-07T08:11:55.4270850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4271187Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4271273Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4271540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4271598Z     layer_outputs = layer(
2025-09-07T08:11:55.4271868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4271978Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4271981Z 
2025-09-07T08:11:55.4272078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4272415Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4272492Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4272758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4272816Z     layer_outputs = layer(
2025-09-07T08:11:55.4273081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4273147Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4273412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4273493Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4273762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1198, in forward
2025-09-07T08:11:55.4273908Z     query_vectors = self.query(hidden_states)
2025-09-07T08:11:55.4273911Z 
2025-09-07T08:11:55.4274000Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4274342Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4274421Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4274692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4274752Z     layer_outputs = layer(
2025-09-07T08:11:55.4275016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4275088Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4275354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4275495Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4275765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1199, in forward
2025-09-07T08:11:55.4275869Z     key_vectors = self.key(hidden_states)
2025-09-07T08:11:55.4275873Z 
2025-09-07T08:11:55.4275954Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4276026Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4276117Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4276455Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4276532Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4276800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4276862Z     layer_outputs = layer(
2025-09-07T08:11:55.4277132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4277195Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4277462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4277540Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4277804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T08:11:55.4277987Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4278271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4278418Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4278422Z 
2025-09-07T08:11:55.4278493Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4278568Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4278662Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4279000Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4279079Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4279346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4279407Z     layer_outputs = layer(
2025-09-07T08:11:55.4279671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4279776Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4280080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4280159Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4280427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T08:11:55.4280594Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4280875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4280943Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4280947Z 
2025-09-07T08:11:55.4281035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4281408Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4281490Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4281792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4281851Z     layer_outputs = layer(
2025-09-07T08:11:55.4282115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4282181Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4282446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4282525Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4282788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T08:11:55.4282935Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4282939Z 
2025-09-07T08:11:55.4283027Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4283362Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4283440Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4283706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4283767Z     layer_outputs = layer(
2025-09-07T08:11:55.4284029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4284094Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4284358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4284437Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4284709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T08:11:55.4284846Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4284849Z 
2025-09-07T08:11:55.4284943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4285280Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4285355Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4285628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4285724Z     layer_outputs = layer(
2025-09-07T08:11:55.4286030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4286095Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4286364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4286440Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4286707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1200, in forward
2025-09-07T08:11:55.4286781Z     value_vectors = self.value(hidden_states)
2025-09-07T08:11:55.4286784Z 
2025-09-07T08:11:55.4286853Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4286924Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4287012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4287381Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4287462Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4287758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4287819Z     layer_outputs = layer(
2025-09-07T08:11:55.4288085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4288151Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4288414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4288490Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4288754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1266, in forward
2025-09-07T08:11:55.4288943Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4289228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4289363Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4289367Z 
2025-09-07T08:11:55.4289438Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4289507Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4289596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4289936Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4290010Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4290280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4290338Z     layer_outputs = layer(
2025-09-07T08:11:55.4290601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4290666Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4290931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4291009Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4291277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T08:11:55.4291387Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4291393Z 
2025-09-07T08:11:55.4291491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4291876Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4291994Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4292272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4292337Z     layer_outputs = layer(
2025-09-07T08:11:55.4292604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4292670Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4292937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4293021Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4293287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T08:11:55.4293439Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4293442Z 
2025-09-07T08:11:55.4293538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4293911Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4293992Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4294261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4294322Z     layer_outputs = layer(
2025-09-07T08:11:55.4294590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4294652Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4294920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4295003Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4295268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1324, in forward
2025-09-07T08:11:55.4295474Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T08:11:55.4295788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T08:11:55.4295926Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T08:11:55.4295929Z 
2025-09-07T08:11:55.4296022Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4296366Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4296458Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4296726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4296794Z     layer_outputs = layer(
2025-09-07T08:11:55.4297062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4297131Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4297396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T08:11:55.4297530Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T08:11:55.4297862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T08:11:55.4297977Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4298010Z 
2025-09-07T08:11:55.4298105Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4298443Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4298522Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4298786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4298844Z     layer_outputs = layer(
2025-09-07T08:11:55.4299111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4299228Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4299495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4299601Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4299868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4299968Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4300259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4300334Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4300612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T08:11:55.4300688Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4300692Z 
2025-09-07T08:11:55.4300782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4301125Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4301209Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4301476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4301536Z     layer_outputs = layer(
2025-09-07T08:11:55.4301799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4301906Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4302175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4302249Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4302516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4302587Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4302881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4302957Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4303374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T08:11:55.4303462Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T08:11:55.4303465Z 
2025-09-07T08:11:55.4303560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4303908Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4303988Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4304363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4304469Z     layer_outputs = layer(
2025-09-07T08:11:55.4304737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4304863Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4305130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4305205Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4305464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4305531Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4305825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T08:11:55.4305897Z     return self.output(hidden_states)
2025-09-07T08:11:55.4306214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T08:11:55.4306291Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4306360Z 
2025-09-07T08:11:55.4306461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4306803Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4306881Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4307157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4307219Z     layer_outputs = layer(
2025-09-07T08:11:55.4307493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4307566Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4307836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4307929Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4308201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 540, in forward
2025-09-07T08:11:55.4308293Z     query_key_vectors = self.query_key(hidden_states)
2025-09-07T08:11:55.4308297Z 
2025-09-07T08:11:55.4308390Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4308735Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4308812Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4309081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4309148Z     layer_outputs = layer(
2025-09-07T08:11:55.4309417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4309491Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4309757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4309837Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4310110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4310250Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4310539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 749, in _hash_vectors
2025-09-07T08:11:55.4310793Z     random_rotations = torch.randn(rotations_shape, device=vectors.device, dtype=vectors.dtype)
2025-09-07T08:11:55.4310797Z 
2025-09-07T08:11:55.4310893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4311235Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4311316Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4311591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4311652Z     layer_outputs = layer(
2025-09-07T08:11:55.4311923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4311989Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4312295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4312382Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4312678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4312817Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4313100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 751, in _hash_vectors
2025-09-07T08:11:55.4313254Z     rotated_vectors = torch.einsum("bmtd,mdhr->bmhtr", vectors, random_rotations)
2025-09-07T08:11:55.4313257Z 
2025-09-07T08:11:55.4313350Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4313708Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4313791Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4314055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4314125Z     layer_outputs = layer(
2025-09-07T08:11:55.4314394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4314466Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4314734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4314814Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4315088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4315222Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4315509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 754, in _hash_vectors
2025-09-07T08:11:55.4315644Z     rotated_vectors = torch.cat([rotated_vectors, -rotated_vectors], dim=-1)
2025-09-07T08:11:55.4315647Z 
2025-09-07T08:11:55.4315746Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4316092Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4316172Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4316450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4316513Z     layer_outputs = layer(
2025-09-07T08:11:55.4316787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4316936Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4317204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4317296Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4317565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T08:11:55.4317708Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T08:11:55.4317994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 755, in _hash_vectors
2025-09-07T08:11:55.4318089Z     buckets = torch.argmax(rotated_vectors, dim=-1)
2025-09-07T08:11:55.4318092Z 
2025-09-07T08:11:55.4318187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4318560Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4318649Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4318949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4319018Z     layer_outputs = layer(
2025-09-07T08:11:55.4319282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4319353Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4319621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4319702Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4319973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.4320167Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.4320541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.4320642Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T08:11:55.4320939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 182, in _stable_argsort
2025-09-07T08:11:55.4321087Z     scaled_vector = vector.shape[dim] * vector + (scale_offset % vector.shape[dim])
2025-09-07T08:11:55.4321090Z 
2025-09-07T08:11:55.4321166Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4321271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4321614Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4321702Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4321971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4322036Z     layer_outputs = layer(
2025-09-07T08:11:55.4322307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4322373Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4322648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4322728Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4322999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T08:11:55.4323248Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T08:11:55.4323613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T08:11:55.4323714Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T08:11:55.4324001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 183, in _stable_argsort
2025-09-07T08:11:55.4324092Z     return torch.argsort(scaled_vector, dim=dim)
2025-09-07T08:11:55.4324095Z 
2025-09-07T08:11:55.4324165Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4324243Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4324336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4324708Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4324798Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4325106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4325177Z     layer_outputs = layer(
2025-09-07T08:11:55.4325445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4325510Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4325784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4325863Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4326136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4326234Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4326501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T08:11:55.4326713Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4326999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4327151Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4327155Z 
2025-09-07T08:11:55.4327227Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4327305Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4327398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4327740Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4327833Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4328102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4328170Z     layer_outputs = layer(
2025-09-07T08:11:55.4328436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4328506Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4328773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4328853Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4329123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4329276Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4329548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T08:11:55.4329743Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4330029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4330107Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4330110Z 
2025-09-07T08:11:55.4330182Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4330282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4330619Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4330707Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4331012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4331077Z     layer_outputs = layer(
2025-09-07T08:11:55.4331384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4331453Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4331727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4331809Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4332075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 541, in forward
2025-09-07T08:11:55.4332158Z     value_vectors = self.value(hidden_states)
2025-09-07T08:11:55.4332167Z 
2025-09-07T08:11:55.4332261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4332611Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4332690Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4332963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4333022Z     layer_outputs = layer(
2025-09-07T08:11:55.4333288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4333359Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4333622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4333704Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4333968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4334061Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4334324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T08:11:55.4334499Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4334782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4334913Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4334917Z 
2025-09-07T08:11:55.4334993Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4335084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4335458Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4335569Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4335840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4335904Z     layer_outputs = layer(
2025-09-07T08:11:55.4336171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4336238Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4336503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4336581Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4336848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4337029Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4337298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T08:11:55.4337495Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4337826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T08:11:55.4337894Z     return torch.cat(slices, dim=3)
2025-09-07T08:11:55.4337898Z 
2025-09-07T08:11:55.4337988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4338328Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4338406Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4338680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4338739Z     layer_outputs = layer(
2025-09-07T08:11:55.4339003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4339067Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4339333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4339413Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4339682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 595, in forward
2025-09-07T08:11:55.4339885Z     query_key_vectors = self._gather_by_expansion(query_key_vectors, sorted_bucket_idx_per_hash, num_hashes)
2025-09-07T08:11:55.4340195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1099, in _gather_by_expansion
2025-09-07T08:11:55.4340280Z     return torch.gather(vectors, 2, expanded_idxs)
2025-09-07T08:11:55.4340283Z 
2025-09-07T08:11:55.4340374Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4340709Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4340792Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4341056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4341116Z     layer_outputs = layer(
2025-09-07T08:11:55.4341382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4341501Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4341806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4341883Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4342152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4342240Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4342500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 849, in _attend
2025-09-07T08:11:55.4342646Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T08:11:55.4342649Z 
2025-09-07T08:11:55.4342741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4343116Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4343197Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4343466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4343555Z     layer_outputs = layer(
2025-09-07T08:11:55.4343826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4343896Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4344164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4344244Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4344509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4344598Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4344868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 845, in _attend
2025-09-07T08:11:55.4345056Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T08:11:55.4345347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T08:11:55.4345481Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T08:11:55.4345491Z 
2025-09-07T08:11:55.4345563Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4345653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4345991Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4346072Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4346343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4346403Z     layer_outputs = layer(
2025-09-07T08:11:55.4346667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4346731Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4346998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4347075Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4347349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4347433Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4347744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T08:11:55.4347885Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4347888Z 
2025-09-07T08:11:55.4347979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4348321Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4348399Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4348665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4348724Z     layer_outputs = layer(
2025-09-07T08:11:55.4348996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4349064Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4349375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4349465Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4349772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T08:11:55.4349873Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T08:11:55.4350142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T08:11:55.4350250Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T08:11:55.4350259Z 
2025-09-07T08:11:55.4350333Z cudagraph partition due to non gpu ops
2025-09-07T08:11:55.4350430Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4350782Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4350869Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4351143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4351208Z     layer_outputs = layer(
2025-09-07T08:11:55.4351473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4351544Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4351809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T08:11:55.4351893Z     self_attention_outputs = self.self_attention(
2025-09-07T08:11:55.4352161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 692, in forward
2025-09-07T08:11:55.4352375Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T08:11:55.4352697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T08:11:55.4352835Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T08:11:55.4352838Z 
2025-09-07T08:11:55.4352938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4353280Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4353361Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4353627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4353726Z     layer_outputs = layer(
2025-09-07T08:11:55.4354028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T08:11:55.4354093Z     attn_outputs = self.attention(
2025-09-07T08:11:55.4354361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T08:11:55.4354494Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T08:11:55.4354759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T08:11:55.4354836Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4354840Z 
2025-09-07T08:11:55.4354929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4355268Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4355381Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4355652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4355741Z     layer_outputs = layer(
2025-09-07T08:11:55.4356011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4356129Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4356392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4356465Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4356728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4356802Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4357092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4357165Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4357433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T08:11:55.4357502Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4357505Z 
2025-09-07T08:11:55.4357596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4357933Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4358011Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4358274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4358334Z     layer_outputs = layer(
2025-09-07T08:11:55.4358606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4358717Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4358986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4359057Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4359311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4359379Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4359660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T08:11:55.4359767Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4360068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T08:11:55.4360146Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T08:11:55.4360149Z 
2025-09-07T08:11:55.4360242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4360580Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4360658Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4360921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4360987Z     layer_outputs = layer(
2025-09-07T08:11:55.4361254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4361405Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4361678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T08:11:55.4361749Z     return apply_chunking_to_forward(
2025-09-07T08:11:55.4362065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:55.4362136Z     return forward_fn(*input_tensors)
2025-09-07T08:11:55.4362429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T08:11:55.4362500Z     return self.output(hidden_states)
2025-09-07T08:11:55.4362769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T08:11:55.4362850Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:11:55.4362856Z 
2025-09-07T08:11:55.4362949Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4363297Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4363377Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4363649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T08:11:55.4363710Z     layer_outputs = layer(
2025-09-07T08:11:55.4363976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T08:11:55.4364094Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T08:11:55.4364097Z 
2025-09-07T08:11:55.4364189Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:55.4364535Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T08:11:55.4364613Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T08:11:55.4364887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1753, in forward
2025-09-07T08:11:55.4364992Z     return torch.cat([attn_output, hidden_states], dim=-1)
2025-09-07T08:11:55.4364995Z 
2025-09-07T08:11:58.5754051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:11:58.5754698Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2495, in torch_dynamo_resume_in_forward_at_2481
2025-09-07T08:11:58.5755232Z     logits = self.lm_head(sequence_output)
2025-09-07T08:11:58.5755677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1886, in forward
2025-09-07T08:11:58.5756609Z     return apply_chunking_to_forward(self.forward_chunk, self.chunk_size_lm_head, self.seq_len_dim, hidden_states)
2025-09-07T08:11:58.5757277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:11:58.5757702Z     return forward_fn(*input_tensors)
2025-09-07T08:11:58.5758131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1889, in forward_chunk
2025-09-07T08:11:58.5758572Z     hidden_states = self.decoder(hidden_states)
2025-09-07T08:11:58.5758718Z 
2025-09-07T08:11:59.4170210Z pass
2025-09-07T08:11:59.4170591Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:01.7739385Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:12:01.7740775Z   import pynvml  # type: ignore[import]
2025-09-07T08:12:03.7821064Z 
2025-09-07T08:12:09.5487796Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:12:09.5488079Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:12:09.5826851Z cpu  eval  hf_Roberta_base                    
2025-09-07T08:12:11.5229556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:12.0577377Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:12.5948140Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:20.0487832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0488281Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0488665Z     return mod(**inputs)
2025-09-07T08:12:20.0489131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0489567Z     outputs = self.roberta(
2025-09-07T08:12:20.0489992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T08:12:20.0490432Z     embedding_output = self.embeddings(
2025-09-07T08:12:20.0490852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 90, in forward
2025-09-07T08:12:20.0491428Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
2025-09-07T08:12:20.0492051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1576, in create_position_ids_from_input_ids
2025-09-07T08:12:20.0492543Z     mask = input_ids.ne(padding_idx).int()
2025-09-07T08:12:20.0492679Z 
2025-09-07T08:12:20.0492769Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0492972Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0493182Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0493373Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0493566Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0493747Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0493968Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0494166Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0494365Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0494548Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0494737Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0494931Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0495160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0495530Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0496282Z     return mod(**inputs)
2025-09-07T08:12:20.0496683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0497100Z     outputs = self.roberta(
2025-09-07T08:12:20.0497490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T08:12:20.0497963Z     embedding_output = self.embeddings(
2025-09-07T08:12:20.0498371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 90, in forward
2025-09-07T08:12:20.0498911Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
2025-09-07T08:12:20.0499515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1577, in create_position_ids_from_input_ids
2025-09-07T08:12:20.0500190Z     incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask
2025-09-07T08:12:20.0500443Z 
2025-09-07T08:12:20.0500552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0500966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0501291Z     return mod(**inputs)
2025-09-07T08:12:20.0501693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0502106Z     outputs = self.roberta(
2025-09-07T08:12:20.0502478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T08:12:20.0502899Z     embedding_output = self.embeddings(
2025-09-07T08:12:20.0503453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 90, in forward
2025-09-07T08:12:20.0504026Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
2025-09-07T08:12:20.0504662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1577, in create_position_ids_from_input_ids
2025-09-07T08:12:20.0505252Z     incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask
2025-09-07T08:12:20.0505500Z 
2025-09-07T08:12:20.0505606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0505960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0506278Z     return mod(**inputs)
2025-09-07T08:12:20.0506663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0507071Z     outputs = self.roberta(
2025-09-07T08:12:20.0507466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T08:12:20.0507881Z     embedding_output = self.embeddings(
2025-09-07T08:12:20.0508283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 120, in forward
2025-09-07T08:12:20.0508706Z     embeddings = self.LayerNorm(embeddings)
2025-09-07T08:12:20.0508839Z 
2025-09-07T08:12:20.0508932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0509282Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0509600Z     return mod(**inputs)
2025-09-07T08:12:20.0509978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0510415Z     outputs = self.roberta(
2025-09-07T08:12:20.0510850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0511308Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0511713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0512123Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0512459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0512830Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0513245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0513659Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0514036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0514409Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0514866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0515278Z     self_outputs = self.self(
2025-09-07T08:12:20.0515684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0516075Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0516496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0517072Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0517347Z 
2025-09-07T08:12:20.0517447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0517814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0518135Z     return mod(**inputs)
2025-09-07T08:12:20.0518521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0518932Z     outputs = self.roberta(
2025-09-07T08:12:20.0519316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0519728Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0520130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0520536Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0520882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0521246Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0521659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0522083Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0522473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0522848Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0523241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0523651Z     self_outputs = self.self(
2025-09-07T08:12:20.0523997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0524360Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0524748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0525230Z     self.key(current_states)
2025-09-07T08:12:20.0525345Z 
2025-09-07T08:12:20.0525448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0525798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0526113Z     return mod(**inputs)
2025-09-07T08:12:20.0526516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0526919Z     outputs = self.roberta(
2025-09-07T08:12:20.0527300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0527708Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0528093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0528496Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0528895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0529263Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0529696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0530116Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0530488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0530856Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0531245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0531642Z     self_outputs = self.self(
2025-09-07T08:12:20.0531992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0532365Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0532746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0533145Z     self.value(current_states)
2025-09-07T08:12:20.0533258Z 
2025-09-07T08:12:20.0533336Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0533565Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0533917Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0534234Z     return mod(**inputs)
2025-09-07T08:12:20.0534621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0535031Z     outputs = self.roberta(
2025-09-07T08:12:20.0535411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0535821Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0536215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0536608Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0536940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0537299Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0537766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0538203Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0538592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0539020Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0539453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0549384Z     self_outputs = self.self(
2025-09-07T08:12:20.0549823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0550227Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0550642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0551128Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0551315Z 
2025-09-07T08:12:20.0551424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0551790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0552116Z     return mod(**inputs)
2025-09-07T08:12:20.0552583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0553008Z     outputs = self.roberta(
2025-09-07T08:12:20.0553430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0553843Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0554239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0554640Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0554977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0555338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0555746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0556171Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0556548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0556914Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0557302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0557757Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0558213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0558629Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0558763Z 
2025-09-07T08:12:20.0558863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0559228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0559551Z     return mod(**inputs)
2025-09-07T08:12:20.0559937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0560344Z     outputs = self.roberta(
2025-09-07T08:12:20.0560718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0561125Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0561525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0561935Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0562266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0562621Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0563077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0563531Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0563934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0564323Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0564763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0565250Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0565700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0566117Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0566247Z 
2025-09-07T08:12:20.0566346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0566733Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0567059Z     return mod(**inputs)
2025-09-07T08:12:20.0567469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0567868Z     outputs = self.roberta(
2025-09-07T08:12:20.0568250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0568669Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0569092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0569503Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0569834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0570195Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0570601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0571018Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0571418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0571804Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0572239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0572718Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0573169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0573609Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0573985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0574318Z     return self.act(input)
2025-09-07T08:12:20.0574430Z 
2025-09-07T08:12:20.0574532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0574887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0575200Z     return mod(**inputs)
2025-09-07T08:12:20.0575581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0575988Z     outputs = self.roberta(
2025-09-07T08:12:20.0576372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0576775Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0577207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0577724Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0578062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0578422Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0578848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0579255Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0579648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0580038Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0580477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0581013Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0581473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0581940Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0582082Z 
2025-09-07T08:12:20.0582341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0582698Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0583017Z     return mod(**inputs)
2025-09-07T08:12:20.0583390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0583798Z     outputs = self.roberta(
2025-09-07T08:12:20.0584182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0584599Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0584986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0585392Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0585729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0586084Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0586486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0586892Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0587270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0587655Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0588045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0588447Z     self_outputs = self.self(
2025-09-07T08:12:20.0588797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0589161Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0589543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0590088Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0590348Z 
2025-09-07T08:12:20.0590446Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0590798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0591162Z     return mod(**inputs)
2025-09-07T08:12:20.0591550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0591997Z     outputs = self.roberta(
2025-09-07T08:12:20.0592374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0592781Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0593177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0593579Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0593912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0594259Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0594663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0595110Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0595485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0595846Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0596264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0596675Z     self_outputs = self.self(
2025-09-07T08:12:20.0597032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0597401Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0597785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0598196Z     self.key(current_states)
2025-09-07T08:12:20.0598313Z 
2025-09-07T08:12:20.0598413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0598771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0599093Z     return mod(**inputs)
2025-09-07T08:12:20.0599481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0599889Z     outputs = self.roberta(
2025-09-07T08:12:20.0600268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0600674Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0601067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0601471Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0601814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0602183Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0602587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0602992Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0603575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0603947Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0604335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0604742Z     self_outputs = self.self(
2025-09-07T08:12:20.0605090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0605546Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0605983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0606385Z     self.value(current_states)
2025-09-07T08:12:20.0606495Z 
2025-09-07T08:12:20.0606573Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0606807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0607162Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0607482Z     return mod(**inputs)
2025-09-07T08:12:20.0607864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0608268Z     outputs = self.roberta(
2025-09-07T08:12:20.0608652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0609057Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0609501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0609907Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0611436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0611810Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0612220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0612633Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0612998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0613368Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0613758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0614168Z     self_outputs = self.self(
2025-09-07T08:12:20.0614514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0614879Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0615262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0615731Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0615915Z 
2025-09-07T08:12:20.0616019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0616365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0616692Z     return mod(**inputs)
2025-09-07T08:12:20.0617075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0617484Z     outputs = self.roberta(
2025-09-07T08:12:20.0617933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0618333Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0618732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0619134Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0619469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0619819Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0620224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0620637Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0621060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0621460Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0621839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0622296Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0622746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0623165Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0623299Z 
2025-09-07T08:12:20.0623401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0623742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0624062Z     return mod(**inputs)
2025-09-07T08:12:20.0624475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0624888Z     outputs = self.roberta(
2025-09-07T08:12:20.0625302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0625707Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0626098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0626500Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0626832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0627179Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0627580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0627998Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0628400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0628794Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0629224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0629710Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0630159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0630571Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0630701Z 
2025-09-07T08:12:20.0630803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0631150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0631471Z     return mod(**inputs)
2025-09-07T08:12:20.0631848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0632255Z     outputs = self.roberta(
2025-09-07T08:12:20.0632630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0633033Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0633423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0633819Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0634143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0634486Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0634940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0635384Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0635771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0636149Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0636566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0637032Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0637463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0637897Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0638303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0638633Z     return self.act(input)
2025-09-07T08:12:20.0638742Z 
2025-09-07T08:12:20.0638834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0639213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0639522Z     return mod(**inputs)
2025-09-07T08:12:20.0639895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0640294Z     outputs = self.roberta(
2025-09-07T08:12:20.0640660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0641053Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0641434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0641828Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0642157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0642497Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0642894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0643297Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0643676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0644062Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0644482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0644970Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0645429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0645850Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0645987Z 
2025-09-07T08:12:20.0646084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0646443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0646762Z     return mod(**inputs)
2025-09-07T08:12:20.0647139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0647548Z     outputs = self.roberta(
2025-09-07T08:12:20.0647928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0648388Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0648822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0649219Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0649561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0649921Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0650316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0650721Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0651085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0651453Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0651829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0652268Z     self_outputs = self.self(
2025-09-07T08:12:20.0652617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0652969Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0653427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0653975Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0654235Z 
2025-09-07T08:12:20.0654331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0654673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0654984Z     return mod(**inputs)
2025-09-07T08:12:20.0655357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0655759Z     outputs = self.roberta(
2025-09-07T08:12:20.0656132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0656536Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0656919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0657316Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0657700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0658055Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0658441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0658849Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0659223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0659584Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0659957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0660356Z     self_outputs = self.self(
2025-09-07T08:12:20.0660690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0661055Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0661429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0661819Z     self.key(current_states)
2025-09-07T08:12:20.0661924Z 
2025-09-07T08:12:20.0662019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0662402Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0662752Z     return mod(**inputs)
2025-09-07T08:12:20.0663118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0663521Z     outputs = self.roberta(
2025-09-07T08:12:20.0663899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0664300Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0664685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0665084Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0665408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0665763Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0666195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0666601Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0666990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0667360Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0667738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0668129Z     self_outputs = self.self(
2025-09-07T08:12:20.0668468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0668818Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0669197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0669607Z     self.value(current_states)
2025-09-07T08:12:20.0669717Z 
2025-09-07T08:12:20.0669793Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0670003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0670348Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0670650Z     return mod(**inputs)
2025-09-07T08:12:20.0671013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0671414Z     outputs = self.roberta(
2025-09-07T08:12:20.0671780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0672174Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0672565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0672959Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0673277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0673620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0674014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0674425Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0674781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0675134Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0675512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0675950Z     self_outputs = self.self(
2025-09-07T08:12:20.0676315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0676679Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0677053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0677501Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0677680Z 
2025-09-07T08:12:20.0677769Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0678104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0678413Z     return mod(**inputs)
2025-09-07T08:12:20.0678793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0679188Z     outputs = self.roberta(
2025-09-07T08:12:20.0679587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0679989Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0680403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0680818Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0681153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0681510Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0681907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0682313Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0682681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0683050Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0683432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0683889Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0684335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0684749Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0684888Z 
2025-09-07T08:12:20.0684981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0685325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0685629Z     return mod(**inputs)
2025-09-07T08:12:20.0686007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0686420Z     outputs = self.roberta(
2025-09-07T08:12:20.0686795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0687200Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0687582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0687980Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0688310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0688661Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0689059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0689517Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0689953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0690347Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0690783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0691282Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0691733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0692145Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0692280Z 
2025-09-07T08:12:20.0692378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0692724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0693035Z     return mod(**inputs)
2025-09-07T08:12:20.0693465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0693871Z     outputs = self.roberta(
2025-09-07T08:12:20.0694281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0694686Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0695073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0695474Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0695811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0696163Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0696562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0696973Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0697370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0697813Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0698242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0698715Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0699158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0699598Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0699971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0700316Z     return self.act(input)
2025-09-07T08:12:20.0700421Z 
2025-09-07T08:12:20.0700517Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0700870Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0701190Z     return mod(**inputs)
2025-09-07T08:12:20.0701567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0701970Z     outputs = self.roberta(
2025-09-07T08:12:20.0702340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0702745Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0703275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0703765Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0704162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0704513Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0704919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0705335Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0705734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0706122Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0706553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0707053Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0707600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0708027Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0708161Z 
2025-09-07T08:12:20.0708258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0708658Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0708982Z     return mod(**inputs)
2025-09-07T08:12:20.0709367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0709773Z     outputs = self.roberta(
2025-09-07T08:12:20.0710150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0710550Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0710947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0711370Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0711712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0712065Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0712466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0712879Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0713253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0713617Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0714013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0714424Z     self_outputs = self.self(
2025-09-07T08:12:20.0714785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0715156Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0715538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0716088Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0716359Z 
2025-09-07T08:12:20.0716457Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0716814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0717134Z     return mod(**inputs)
2025-09-07T08:12:20.0717508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0717965Z     outputs = self.roberta(
2025-09-07T08:12:20.0718410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0718819Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0719217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0719619Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0719965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0720326Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0720733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0721141Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0721522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0721933Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0722323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0722756Z     self_outputs = self.self(
2025-09-07T08:12:20.0723102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0723464Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0723844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0724247Z     self.key(current_states)
2025-09-07T08:12:20.0724354Z 
2025-09-07T08:12:20.0724451Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0724798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0725113Z     return mod(**inputs)
2025-09-07T08:12:20.0725492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0725894Z     outputs = self.roberta(
2025-09-07T08:12:20.0726268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0726673Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0727066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0727466Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0727800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0728147Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0728550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0728964Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0729333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0729690Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0730074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0730474Z     self_outputs = self.self(
2025-09-07T08:12:20.0730822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0731182Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0731560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0732000Z     self.value(current_states)
2025-09-07T08:12:20.0732148Z 
2025-09-07T08:12:20.0732224Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0732449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0732795Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0733105Z     return mod(**inputs)
2025-09-07T08:12:20.0733484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0733893Z     outputs = self.roberta(
2025-09-07T08:12:20.0734270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0734666Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0735062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0735468Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0735842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0736200Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0736630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0737053Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0737426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0737846Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0738241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0738649Z     self_outputs = self.self(
2025-09-07T08:12:20.0739007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0739375Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0739759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0740215Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0740403Z 
2025-09-07T08:12:20.0740503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0740858Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0741175Z     return mod(**inputs)
2025-09-07T08:12:20.0741555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0741957Z     outputs = self.roberta(
2025-09-07T08:12:20.0742335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0742743Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0743134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0743539Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0743867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0744219Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0744622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0745033Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0745395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0745833Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0746256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0746712Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0747166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0747582Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0747722Z 
2025-09-07T08:12:20.0747816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0748159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0748477Z     return mod(**inputs)
2025-09-07T08:12:20.0748854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0749253Z     outputs = self.roberta(
2025-09-07T08:12:20.0749680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0750088Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0750520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0750922Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0751268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0751617Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0752019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0752434Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0752826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0753230Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0753666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0754152Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0754602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0755014Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0755154Z 
2025-09-07T08:12:20.0755249Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0755594Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0755912Z     return mod(**inputs)
2025-09-07T08:12:20.0756287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0756689Z     outputs = self.roberta(
2025-09-07T08:12:20.0757067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0757473Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0757866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0758264Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0758598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0758952Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0759352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0759811Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0760248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0760645Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0761077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0761557Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0762001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0762439Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0762817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0763155Z     return self.act(input)
2025-09-07T08:12:20.0763261Z 
2025-09-07T08:12:20.0763364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0763751Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0764066Z     return mod(**inputs)
2025-09-07T08:12:20.0764486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0764903Z     outputs = self.roberta(
2025-09-07T08:12:20.0765285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0765687Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0766079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0766486Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0766821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0767180Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0767573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0767985Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0768379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0768771Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0769205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0769695Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0770166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0770591Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0770731Z 
2025-09-07T08:12:20.0770841Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0771198Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0771524Z     return mod(**inputs)
2025-09-07T08:12:20.0771912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0772321Z     outputs = self.roberta(
2025-09-07T08:12:20.0772708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0773114Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0773515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0773970Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0774346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0774713Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0775117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0775532Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0775916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0776293Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0776683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0777077Z     self_outputs = self.self(
2025-09-07T08:12:20.0777430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0777885Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0778274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0778844Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0779119Z 
2025-09-07T08:12:20.0779219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0779577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0779898Z     return mod(**inputs)
2025-09-07T08:12:20.0780280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0780683Z     outputs = self.roberta(
2025-09-07T08:12:20.0781072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0781479Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0781880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0782288Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0782621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0782974Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0783396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0783823Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0784202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0784582Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0784983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0785391Z     self_outputs = self.self(
2025-09-07T08:12:20.0785749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0786109Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0786491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0786902Z     self.key(current_states)
2025-09-07T08:12:20.0787009Z 
2025-09-07T08:12:20.0787117Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0787476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0787792Z     return mod(**inputs)
2025-09-07T08:12:20.0788229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0788677Z     outputs = self.roberta(
2025-09-07T08:12:20.0789067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0789468Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0789872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0790278Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0790620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0790985Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0791386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0792054Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0792434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0792807Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0793228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0793630Z     self_outputs = self.self(
2025-09-07T08:12:20.0793982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0794348Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0794728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0795241Z     self.value(current_states)
2025-09-07T08:12:20.0795361Z 
2025-09-07T08:12:20.0795437Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0795674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0796026Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0796344Z     return mod(**inputs)
2025-09-07T08:12:20.0796726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0797131Z     outputs = self.roberta(
2025-09-07T08:12:20.0797510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0797914Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0798309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0798703Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0799046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0799408Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0799814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0800222Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0800595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0800960Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0801345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0801746Z     self_outputs = self.self(
2025-09-07T08:12:20.0802084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0802517Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0802937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0803537Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0803721Z 
2025-09-07T08:12:20.0803828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0804172Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0804484Z     return mod(**inputs)
2025-09-07T08:12:20.0804866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0805272Z     outputs = self.roberta(
2025-09-07T08:12:20.0805646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0806051Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0806528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0806944Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0807355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0807717Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0808131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0808545Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0808926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0809297Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0809683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0810148Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0810607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0811029Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0811166Z 
2025-09-07T08:12:20.0811268Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0811611Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0811928Z     return mod(**inputs)
2025-09-07T08:12:20.0812311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0812718Z     outputs = self.roberta(
2025-09-07T08:12:20.0813091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0813502Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0813902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0814314Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0814650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0815002Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0815406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0815828Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0816229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0816702Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0817179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0817721Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0818174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0818591Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0818722Z 
2025-09-07T08:12:20.0818817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0819164Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0819484Z     return mod(**inputs)
2025-09-07T08:12:20.0819864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0820274Z     outputs = self.roberta(
2025-09-07T08:12:20.0820683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0821094Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0821519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0821936Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0822285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0822637Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0823040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0823449Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0823842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0824228Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0824665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0825137Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0825572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0826006Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0826369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0826699Z     return self.act(input)
2025-09-07T08:12:20.0826803Z 
2025-09-07T08:12:20.0826895Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0827240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0827549Z     return mod(**inputs)
2025-09-07T08:12:20.0827918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0828319Z     outputs = self.roberta(
2025-09-07T08:12:20.0828691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0829091Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0829480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0829874Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0830206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0830603Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0831039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0831449Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0831843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0840082Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0840584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0841114Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0841582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0842013Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0842161Z 
2025-09-07T08:12:20.0842359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0842726Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0843047Z     return mod(**inputs)
2025-09-07T08:12:20.0843477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0843893Z     outputs = self.roberta(
2025-09-07T08:12:20.0844287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0844703Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0845096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0845494Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0845844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0846212Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0846622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0847035Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0847409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0847775Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0848164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0848561Z     self_outputs = self.self(
2025-09-07T08:12:20.0848905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0849261Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0849665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0850217Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0850481Z 
2025-09-07T08:12:20.0850594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0850939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0851262Z     return mod(**inputs)
2025-09-07T08:12:20.0851637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0852033Z     outputs = self.roberta(
2025-09-07T08:12:20.0852405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0852908Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0853303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0853700Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0854034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0854385Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0854779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0855182Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0855548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0855911Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0856327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0856734Z     self_outputs = self.self(
2025-09-07T08:12:20.0857082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0857474Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0857919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0858318Z     self.key(current_states)
2025-09-07T08:12:20.0858430Z 
2025-09-07T08:12:20.0858524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0858873Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0859185Z     return mod(**inputs)
2025-09-07T08:12:20.0859555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0859965Z     outputs = self.roberta(
2025-09-07T08:12:20.0860338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0860758Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0861168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0861569Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0861906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0862263Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0862668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0863079Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0863449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0863814Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0864207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0864606Z     self_outputs = self.self(
2025-09-07T08:12:20.0864951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0865309Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0865685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0866082Z     self.value(current_states)
2025-09-07T08:12:20.0866191Z 
2025-09-07T08:12:20.0866273Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0866541Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0866936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0867253Z     return mod(**inputs)
2025-09-07T08:12:20.0867628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0868021Z     outputs = self.roberta(
2025-09-07T08:12:20.0868391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0868791Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0869179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0869578Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0869909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0870295Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0870700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0871115Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0871539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0871910Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0872300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0872697Z     self_outputs = self.self(
2025-09-07T08:12:20.0873044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0873401Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0873781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0874240Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0874420Z 
2025-09-07T08:12:20.0874518Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0874864Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0875173Z     return mod(**inputs)
2025-09-07T08:12:20.0875550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0875957Z     outputs = self.roberta(
2025-09-07T08:12:20.0876335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0876740Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0877130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0877532Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0877870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0878227Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0878629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0879036Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0879401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0879767Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0880148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0880644Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0881129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0881536Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0881669Z 
2025-09-07T08:12:20.0881766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0882108Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0882412Z     return mod(**inputs)
2025-09-07T08:12:20.0882779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0883179Z     outputs = self.roberta(
2025-09-07T08:12:20.0883547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0883945Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0884412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0884813Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0885177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0885526Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0885916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0886322Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0886712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0887096Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0887525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0888004Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0888445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0888848Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0888976Z 
2025-09-07T08:12:20.0889072Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0889409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0889712Z     return mod(**inputs)
2025-09-07T08:12:20.0890080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0890476Z     outputs = self.roberta(
2025-09-07T08:12:20.0890848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0891241Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0891626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0892017Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0892344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0892696Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0893086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0893493Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0893885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0894343Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0894771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0895245Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0895686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0896124Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0896501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0896835Z     return self.act(input)
2025-09-07T08:12:20.0896936Z 
2025-09-07T08:12:20.0897033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0897378Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0897730Z     return mod(**inputs)
2025-09-07T08:12:20.0898149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0898557Z     outputs = self.roberta(
2025-09-07T08:12:20.0898966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0899379Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0899761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0900152Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0900476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0900821Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0901214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0901291Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0901549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0901617Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0901924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0902047Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0902317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0902386Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0902390Z 
2025-09-07T08:12:20.0902486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0902679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0902738Z     return mod(**inputs)
2025-09-07T08:12:20.0903015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0903237Z     outputs = self.roberta(
2025-09-07T08:12:20.0903529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0903594Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0903867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0903927Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0904144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0904332Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0904654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0904731Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0904978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0905042Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0905313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0905375Z     self_outputs = self.self(
2025-09-07T08:12:20.0905611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0905668Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0905987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0906200Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0906204Z 
2025-09-07T08:12:20.0906342Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0906545Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0906606Z     return mod(**inputs)
2025-09-07T08:12:20.0906895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0906958Z     outputs = self.roberta(
2025-09-07T08:12:20.0907233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0907297Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0907569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0907635Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0907854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0907925Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0908197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0908277Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0908519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0908581Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0908855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0908920Z     self_outputs = self.self(
2025-09-07T08:12:20.0909160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0909219Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0909492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0909556Z     self.key(current_states)
2025-09-07T08:12:20.0909559Z 
2025-09-07T08:12:20.0909654Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0909849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0909907Z     return mod(**inputs)
2025-09-07T08:12:20.0910185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0910286Z     outputs = self.roberta(
2025-09-07T08:12:20.0910593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0910658Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0910931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0910994Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0911210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0911277Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0911552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0911624Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0911864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0911980Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0912254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0912348Z     self_outputs = self.self(
2025-09-07T08:12:20.0912591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0912657Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0912927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0912992Z     self.value(current_states)
2025-09-07T08:12:20.0912995Z 
2025-09-07T08:12:20.0913068Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0913160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0913354Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0913412Z     return mod(**inputs)
2025-09-07T08:12:20.0913687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0913746Z     outputs = self.roberta(
2025-09-07T08:12:20.0914016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0914083Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0914356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0914426Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0914641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0914711Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0914990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0915062Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0915308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0915368Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0915643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0915703Z     self_outputs = self.self(
2025-09-07T08:12:20.0915940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0916005Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0916278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0916481Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0916484Z 
2025-09-07T08:12:20.0916573Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0916759Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0916816Z     return mod(**inputs)
2025-09-07T08:12:20.0917091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0917150Z     outputs = self.roberta(
2025-09-07T08:12:20.0917420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0917488Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0917758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0917825Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0918074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0918142Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0918452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0918521Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0918754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0918814Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0919082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0919203Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0919474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0919554Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0919558Z 
2025-09-07T08:12:20.0919649Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0919833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0919894Z     return mod(**inputs)
2025-09-07T08:12:20.0920168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0920229Z     outputs = self.roberta(
2025-09-07T08:12:20.0920500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0920562Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0920837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0920898Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0921116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0921181Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0921449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0921527Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0921788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0921859Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0922166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0922344Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0922614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0922686Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0922690Z 
2025-09-07T08:12:20.0922780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0922966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0923028Z     return mod(**inputs)
2025-09-07T08:12:20.0923302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0923359Z     outputs = self.roberta(
2025-09-07T08:12:20.0923638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0923707Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0924018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0924081Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0924331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0924399Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0924671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0924748Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0925005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0925074Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0925381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0925488Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0925765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0925867Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0926079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0926141Z     return self.act(input)
2025-09-07T08:12:20.0926144Z 
2025-09-07T08:12:20.0926239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0926423Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0926477Z     return mod(**inputs)
2025-09-07T08:12:20.0926761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0926822Z     outputs = self.roberta(
2025-09-07T08:12:20.0927098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0927161Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0927438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0927506Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0927718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0927789Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0928057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0928195Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0928454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0928519Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0928825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0928948Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0929220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0929293Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0929296Z 
2025-09-07T08:12:20.0929389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0929578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0929664Z     return mod(**inputs)
2025-09-07T08:12:20.0929958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0930018Z     outputs = self.roberta(
2025-09-07T08:12:20.0930321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0930385Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0930657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0930726Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0930939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0931005Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0931276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0931347Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0931589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0931649Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0931920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0931980Z     self_outputs = self.self(
2025-09-07T08:12:20.0932219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0932280Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0932548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0932757Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0932760Z 
2025-09-07T08:12:20.0932851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0933043Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0933100Z     return mod(**inputs)
2025-09-07T08:12:20.0933375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0933439Z     outputs = self.roberta(
2025-09-07T08:12:20.0933708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0933772Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0934042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0934401Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0934617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0934682Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0934955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0935024Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0935261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0935317Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0935584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0935649Z     self_outputs = self.self(
2025-09-07T08:12:20.0935925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0935992Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0936298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0936359Z     self.key(current_states)
2025-09-07T08:12:20.0936364Z 
2025-09-07T08:12:20.0936460Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0936652Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0936712Z     return mod(**inputs)
2025-09-07T08:12:20.0936990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0937057Z     outputs = self.roberta(
2025-09-07T08:12:20.0937331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0937400Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0937723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0937788Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0938011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0938079Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0938350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0938423Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0938658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0938723Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0938996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0939058Z     self_outputs = self.self(
2025-09-07T08:12:20.0939292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0939350Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0939621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0939683Z     self.value(current_states)
2025-09-07T08:12:20.0939686Z 
2025-09-07T08:12:20.0939759Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0939848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0940033Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0940132Z     return mod(**inputs)
2025-09-07T08:12:20.0940442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0940504Z     outputs = self.roberta(
2025-09-07T08:12:20.0940775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0940837Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0941110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0941172Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0941388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0941454Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0941724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0941828Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0942070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0942164Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0942439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0942506Z     self_outputs = self.self(
2025-09-07T08:12:20.0942746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0942808Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0943086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0943209Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0943214Z 
2025-09-07T08:12:20.0943316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0943504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0943568Z     return mod(**inputs)
2025-09-07T08:12:20.0943846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0943903Z     outputs = self.roberta(
2025-09-07T08:12:20.0944179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0944243Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0944513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0944575Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0944794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0944868Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0945136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0945209Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0945445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0945503Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0945770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0945886Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0946157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0946293Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0946296Z 
2025-09-07T08:12:20.0946387Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0946572Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0946627Z     return mod(**inputs)
2025-09-07T08:12:20.0946904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0946961Z     outputs = self.roberta(
2025-09-07T08:12:20.0947233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0947295Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0947563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0947629Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0947874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0947944Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0948247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0948321Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0948586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0948655Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0948966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0949079Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0949352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0949420Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0949424Z 
2025-09-07T08:12:20.0949515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0949698Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0949753Z     return mod(**inputs)
2025-09-07T08:12:20.0950029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0950087Z     outputs = self.roberta(
2025-09-07T08:12:20.0950356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0950416Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0950688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0950752Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0950965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0951032Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0951299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0951370Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0951631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0951696Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0952001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0952164Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0952437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0952538Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0952746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0952807Z     return self.act(input)
2025-09-07T08:12:20.0952811Z 
2025-09-07T08:12:20.0952901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0953086Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0953143Z     return mod(**inputs)
2025-09-07T08:12:20.0953421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0953486Z     outputs = self.roberta(
2025-09-07T08:12:20.0953790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0953860Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0954166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0954231Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0954445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0954513Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0954786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0954854Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0955119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0955183Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0955490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0955616Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0955888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0955959Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0955963Z 
2025-09-07T08:12:20.0956049Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0956235Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0956290Z     return mod(**inputs)
2025-09-07T08:12:20.0956564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0956627Z     outputs = self.roberta(
2025-09-07T08:12:20.0956899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0956962Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0957229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0957287Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0957500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0957564Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0957834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0957977Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0958221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0958282Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0958556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0958631Z     self_outputs = self.self(
2025-09-07T08:12:20.0958871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0958936Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0959206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0959409Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0959422Z 
2025-09-07T08:12:20.0959546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0959733Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0959836Z     return mod(**inputs)
2025-09-07T08:12:20.0960115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0960185Z     outputs = self.roberta(
2025-09-07T08:12:20.0960457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0960523Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0960801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0960864Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0961093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0961162Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0961437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0961522Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0961761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0961832Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0962107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0962180Z     self_outputs = self.self(
2025-09-07T08:12:20.0962419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0962482Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0962760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0962820Z     self.key(current_states)
2025-09-07T08:12:20.0962825Z 
2025-09-07T08:12:20.0962923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0963109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0963167Z     return mod(**inputs)
2025-09-07T08:12:20.0963447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0963507Z     outputs = self.roberta(
2025-09-07T08:12:20.0963783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0963880Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0964184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0964254Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0964470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0964543Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0964812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0964892Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0965128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0965191Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0965472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0965567Z     self_outputs = self.self(
2025-09-07T08:12:20.0965816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0965905Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0966179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0966250Z     self.value(current_states)
2025-09-07T08:12:20.0966253Z 
2025-09-07T08:12:20.0966327Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0966425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0966613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0966670Z     return mod(**inputs)
2025-09-07T08:12:20.0966951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0967014Z     outputs = self.roberta(
2025-09-07T08:12:20.0967295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0967359Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0967635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0967697Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0967909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0967981Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0968252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0968331Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0968569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0968630Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0968909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0968969Z     self_outputs = self.self(
2025-09-07T08:12:20.0969212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0969274Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0969550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0969675Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0969709Z 
2025-09-07T08:12:20.0969799Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0970021Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0970079Z     return mod(**inputs)
2025-09-07T08:12:20.0970364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0970423Z     outputs = self.roberta(
2025-09-07T08:12:20.0970694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0970764Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0971035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0971105Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0971317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0971425Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0971699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0971803Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0972056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0972119Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0972402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0972522Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0972796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0972884Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0972889Z 
2025-09-07T08:12:20.0972984Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0973182Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0973239Z     return mod(**inputs)
2025-09-07T08:12:20.0973533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0973594Z     outputs = self.roberta(
2025-09-07T08:12:20.0973868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0973942Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0974217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0974290Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0974511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0974586Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0974866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0974942Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0975211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0975281Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0975593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0975717Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0976002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.0976147Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0976150Z 
2025-09-07T08:12:20.0976250Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0976455Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0976515Z     return mod(**inputs)
2025-09-07T08:12:20.0976798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0976866Z     outputs = self.roberta(
2025-09-07T08:12:20.0977137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0977208Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0977479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0977639Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0977872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0977947Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0978277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0978356Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0978624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0978695Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0979004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.0979128Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.0979406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.0979517Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.0979730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.0979794Z     return self.act(input)
2025-09-07T08:12:20.0979802Z 
2025-09-07T08:12:20.0979903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0980096Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0980161Z     return mod(**inputs)
2025-09-07T08:12:20.0980440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0980509Z     outputs = self.roberta(
2025-09-07T08:12:20.0980785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0980853Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0981132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0981197Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0981421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0981493Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0981766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.0981848Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.0982107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.0982240Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.0982546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.0982683Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.0982955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.0983027Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0983031Z 
2025-09-07T08:12:20.0983132Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0983321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0983385Z     return mod(**inputs)
2025-09-07T08:12:20.0983661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0983755Z     outputs = self.roberta(
2025-09-07T08:12:20.0984039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0984132Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0984413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0984476Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0984699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0984769Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0985050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0985134Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0985381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0985451Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0985723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0985788Z     self_outputs = self.self(
2025-09-07T08:12:20.0986034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0986094Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0986373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.0986575Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.0986580Z 
2025-09-07T08:12:20.0986670Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0986863Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0986918Z     return mod(**inputs)
2025-09-07T08:12:20.0987198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0987258Z     outputs = self.roberta(
2025-09-07T08:12:20.0987532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0987594Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0987864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0987927Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0988140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0988316Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0988589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0988665Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0988902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0988964Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0989237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0989298Z     self_outputs = self.self(
2025-09-07T08:12:20.0989534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0989593Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0989896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.0989964Z     self.key(current_states)
2025-09-07T08:12:20.0989967Z 
2025-09-07T08:12:20.0990061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0990286Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0990342Z     return mod(**inputs)
2025-09-07T08:12:20.0990618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0990681Z     outputs = self.roberta(
2025-09-07T08:12:20.0990950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0991014Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0991286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0991356Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0991570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0991636Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0991909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0991981Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0992219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0992278Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0992546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0992616Z     self_outputs = self.self(
2025-09-07T08:12:20.0992852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0992912Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0993182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.0993250Z     self.value(current_states)
2025-09-07T08:12:20.0993253Z 
2025-09-07T08:12:20.0993325Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.0993413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0993599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0993654Z     return mod(**inputs)
2025-09-07T08:12:20.0993932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0994024Z     outputs = self.roberta(
2025-09-07T08:12:20.0994325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0994396Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0994669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0994732Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0994945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0995011Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0995285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0995353Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0995623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0995685Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0995960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.0996052Z     self_outputs = self.self(
2025-09-07T08:12:20.0996289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0996350Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0996621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.0996750Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.0996753Z 
2025-09-07T08:12:20.0996842Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.0997026Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.0997089Z     return mod(**inputs)
2025-09-07T08:12:20.0997363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.0997426Z     outputs = self.roberta(
2025-09-07T08:12:20.0997693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.0997758Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.0998027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.0998085Z     layer_outputs = layer_module(
2025-09-07T08:12:20.0998298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.0998364Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.0998639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.0998708Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.0998942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.0999003Z     return func(*args, **kwargs)
2025-09-07T08:12:20.0999271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.0999394Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.0999665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.0999742Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.0999778Z 
2025-09-07T08:12:20.0999868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1000083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1000140Z     return mod(**inputs)
2025-09-07T08:12:20.1000418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1000480Z     outputs = self.roberta(
2025-09-07T08:12:20.1000751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1000812Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1001086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1001145Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1001359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1001475Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1001747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1001849Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1002113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1002186Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1002491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.1002608Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.1002878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.1002953Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1002958Z 
2025-09-07T08:12:20.1003204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1003391Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1003452Z     return mod(**inputs)
2025-09-07T08:12:20.1003730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1003789Z     outputs = self.roberta(
2025-09-07T08:12:20.1004063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1004125Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1004408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1004470Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1004686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1004751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1005020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1005091Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1005348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1005415Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1005715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.1005817Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.1006152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.1006297Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.1006511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.1006571Z     return self.act(input)
2025-09-07T08:12:20.1006575Z 
2025-09-07T08:12:20.1006674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1006865Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1006923Z     return mod(**inputs)
2025-09-07T08:12:20.1007212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1007274Z     outputs = self.roberta(
2025-09-07T08:12:20.1007548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1007659Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1007933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1008038Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1008259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1008330Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1008599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1008674Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1008936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1009004Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1009314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.1009440Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.1009715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.1009787Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1009790Z 
2025-09-07T08:12:20.1009884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1010067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1010123Z     return mod(**inputs)
2025-09-07T08:12:20.1010398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1010457Z     outputs = self.roberta(
2025-09-07T08:12:20.1010733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1010793Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1011062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1011126Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1011341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1011410Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1011681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1011756Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1012006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1012136Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1012417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1012483Z     self_outputs = self.self(
2025-09-07T08:12:20.1012728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1012789Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1013063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.1013275Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.1013279Z 
2025-09-07T08:12:20.1013370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1013595Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1013656Z     return mod(**inputs)
2025-09-07T08:12:20.1013941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1014039Z     outputs = self.roberta(
2025-09-07T08:12:20.1014317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1014389Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1014662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1014732Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1014949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1015018Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1015301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1015370Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1015620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1015682Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1015958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1016025Z     self_outputs = self.self(
2025-09-07T08:12:20.1016260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1016329Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1016598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.1016666Z     self.key(current_states)
2025-09-07T08:12:20.1016674Z 
2025-09-07T08:12:20.1016767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1016959Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1017023Z     return mod(**inputs)
2025-09-07T08:12:20.1017315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1017381Z     outputs = self.roberta(
2025-09-07T08:12:20.1017705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1017770Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1018050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1018174Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1018408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1018480Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1018760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1018840Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1019079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1019148Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1019418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1019487Z     self_outputs = self.self(
2025-09-07T08:12:20.1019756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1019818Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1020096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.1020189Z     self.value(current_states)
2025-09-07T08:12:20.1020192Z 
2025-09-07T08:12:20.1020273Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.1020374Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1020572Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1020636Z     return mod(**inputs)
2025-09-07T08:12:20.1020911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1020976Z     outputs = self.roberta(
2025-09-07T08:12:20.1021249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1021314Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1021587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1021651Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1021872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1021939Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1022214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1022286Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1022521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1022586Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1022861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1022927Z     self_outputs = self.self(
2025-09-07T08:12:20.1023166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1023227Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1023502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.1023626Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.1023630Z 
2025-09-07T08:12:20.1023724Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1023910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1024004Z     return mod(**inputs)
2025-09-07T08:12:20.1024331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1024392Z     outputs = self.roberta(
2025-09-07T08:12:20.1024675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1024737Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1025010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1025074Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1025289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1025362Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1025630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1025758Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1026001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1026092Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1026372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.1026491Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.1026767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.1026839Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1026842Z 
2025-09-07T08:12:20.1026940Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1027130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1027190Z     return mod(**inputs)
2025-09-07T08:12:20.1027467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1027530Z     outputs = self.roberta(
2025-09-07T08:12:20.1027807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1027870Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1028137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1028201Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1028413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1028486Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1028756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1028832Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1029098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1029165Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1029473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.1029583Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.1029859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.1029931Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1029966Z 
2025-09-07T08:12:20.1030058Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1030274Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1030333Z     return mod(**inputs)
2025-09-07T08:12:20.1030620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1030686Z     outputs = self.roberta(
2025-09-07T08:12:20.1030961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1031026Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1031293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1031361Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1031573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1031676Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1031950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1032054Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1032324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1032391Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1032707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.1032812Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.1033104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.1033209Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.1033418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.1033485Z     return self.act(input)
2025-09-07T08:12:20.1033489Z 
2025-09-07T08:12:20.1033579Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1033769Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1033824Z     return mod(**inputs)
2025-09-07T08:12:20.1034101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1034170Z     outputs = self.roberta(
2025-09-07T08:12:20.1034443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1034516Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1034790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1034860Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1035074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1035142Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1035417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1035488Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1035750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1035814Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1036118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.1036310Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.1036581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.1036658Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1036661Z 
2025-09-07T08:12:20.1036751Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1036940Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1036995Z     return mod(**inputs)
2025-09-07T08:12:20.1037270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1037337Z     outputs = self.roberta(
2025-09-07T08:12:20.1037640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1037710Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1037978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1038071Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1038291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1038360Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1038638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1038709Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1038952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1039015Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1039289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1039355Z     self_outputs = self.self(
2025-09-07T08:12:20.1039593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1039653Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1039923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T08:12:20.1040126Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T08:12:20.1040135Z 
2025-09-07T08:12:20.1040228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1040414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1040478Z     return mod(**inputs)
2025-09-07T08:12:20.1040755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1040820Z     outputs = self.roberta(
2025-09-07T08:12:20.1041092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1041156Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1041429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1041492Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1041709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1041776Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1042086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1042192Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1042429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1042498Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1042771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1042834Z     self_outputs = self.self(
2025-09-07T08:12:20.1043071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1043131Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1043408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T08:12:20.1043472Z     self.key(current_states)
2025-09-07T08:12:20.1043477Z 
2025-09-07T08:12:20.1043602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1043785Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1043844Z     return mod(**inputs)
2025-09-07T08:12:20.1044158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1044217Z     outputs = self.roberta(
2025-09-07T08:12:20.1044492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1044556Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1044827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1044896Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1045115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1045189Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1045461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1045535Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1045769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1045830Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1046104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1046164Z     self_outputs = self.self(
2025-09-07T08:12:20.1046404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1046465Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1046737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T08:12:20.1046802Z     self.value(current_states)
2025-09-07T08:12:20.1046805Z 
2025-09-07T08:12:20.1046879Z cudagraph partition due to non gpu ops
2025-09-07T08:12:20.1046977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1047157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1047215Z     return mod(**inputs)
2025-09-07T08:12:20.1047496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1047557Z     outputs = self.roberta(
2025-09-07T08:12:20.1047837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1047975Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1048254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1048314Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1048527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1048600Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1048869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1048947Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1049183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1049244Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1049555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T08:12:20.1049617Z     self_outputs = self.self(
2025-09-07T08:12:20.1049890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1049949Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1050227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T08:12:20.1050350Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:12:20.1050353Z 
2025-09-07T08:12:20.1050443Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1050634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1050691Z     return mod(**inputs)
2025-09-07T08:12:20.1050976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1051036Z     outputs = self.roberta(
2025-09-07T08:12:20.1051307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1051377Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1051648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1051716Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1051933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1052005Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1052272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T08:12:20.1052345Z     self_attention_outputs = self.attention(
2025-09-07T08:12:20.1052590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T08:12:20.1052649Z     return func(*args, **kwargs)
2025-09-07T08:12:20.1052927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T08:12:20.1053045Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T08:12:20.1053323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T08:12:20.1053399Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1053403Z 
2025-09-07T08:12:20.1053489Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1053678Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1053769Z     return mod(**inputs)
2025-09-07T08:12:20.1054079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1054139Z     outputs = self.roberta(
2025-09-07T08:12:20.1054407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1054474Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1054745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1054812Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1055025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1055090Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1055367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1055474Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1055743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1055855Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1056166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.1056282Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.1056554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T08:12:20.1056633Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1056636Z 
2025-09-07T08:12:20.1056727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1056919Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1056977Z     return mod(**inputs)
2025-09-07T08:12:20.1057252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1057318Z     outputs = self.roberta(
2025-09-07T08:12:20.1057642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1057712Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1057983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1058044Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1058262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1058329Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1058611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1058681Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1058940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1059004Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1059305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T08:12:20.1059416Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T08:12:20.1059686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T08:12:20.1059791Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T08:12:20.1060071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:12:20.1060135Z     return self.act(input)
2025-09-07T08:12:20.1060141Z 
2025-09-07T08:12:20.1060232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1060420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1060482Z     return mod(**inputs)
2025-09-07T08:12:20.1060756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T08:12:20.1060822Z     outputs = self.roberta(
2025-09-07T08:12:20.1061091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T08:12:20.1061155Z     encoder_outputs = self.encoder(
2025-09-07T08:12:20.1061461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T08:12:20.1061522Z     layer_outputs = layer_module(
2025-09-07T08:12:20.1061736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:20.1061829Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:20.1062104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T08:12:20.1062181Z     layer_output = apply_chunking_to_forward(
2025-09-07T08:12:20.1062436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T08:12:20.1062507Z     return forward_fn(*input_tensors)
2025-09-07T08:12:20.1062806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T08:12:20.1062937Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T08:12:20.1063206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T08:12:20.1063276Z     hidden_states = self.dense(hidden_states)
2025-09-07T08:12:20.1063279Z 
2025-09-07T08:12:20.1063370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1063553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1063611Z     return mod(**inputs)
2025-09-07T08:12:20.1063883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1108, in forward
2025-09-07T08:12:20.1063970Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T08:12:20.1064242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1143, in forward
2025-09-07T08:12:20.1064306Z     x = self.dense(features)
2025-09-07T08:12:20.1064310Z 
2025-09-07T08:12:20.1064406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:20.1064589Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:20.1064645Z     return mod(**inputs)
2025-09-07T08:12:20.1064915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1108, in forward
2025-09-07T08:12:20.1064999Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T08:12:20.1065279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1148, in forward
2025-09-07T08:12:20.1065337Z     x = self.decoder(x)
2025-09-07T08:12:20.1065340Z 
2025-09-07T08:12:31.2481510Z pass
2025-09-07T08:12:31.2481909Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:34.7636828Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:12:34.7637693Z   import pynvml  # type: ignore[import]
2025-09-07T08:12:36.7731013Z 
2025-09-07T08:12:39.6209580Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:12:39.6210127Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:12:39.6429659Z cpu  eval  hf_T5                              
2025-09-07T08:12:40.6466727Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:41.1044304Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:41.5519672Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:12:53.3577254Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3577975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3578384Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3578736Z     return mod(**inputs)
2025-09-07T08:12:53.3579215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3587379Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3587853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3588269Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3588638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3589013Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3589412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3589833Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3590216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3590597Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3590965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
2025-09-07T08:12:53.3591344Z     position_bias = position_bias + causal_mask
2025-09-07T08:12:53.3591487Z 
2025-09-07T08:12:53.3591594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3591967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3592297Z     return mod(**inputs)
2025-09-07T08:12:53.3592648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3593031Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3593390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3593760Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3594114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3594496Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3594868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3595245Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3595620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.3596028Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3596542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3596986Z     return self.weight * hidden_states
2025-09-07T08:12:53.3597126Z 
2025-09-07T08:12:53.3597232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3597603Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3597932Z     return mod(**inputs)
2025-09-07T08:12:53.3598290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3598664Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3599034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3599407Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3599759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3600195Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3600587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3600962Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3601360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3601743Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3602122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.3602493Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.3602629Z 
2025-09-07T08:12:53.3602730Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3603246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3603576Z     return mod(**inputs)
2025-09-07T08:12:53.3603932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3604314Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3604682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3605054Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3605539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3605894Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3606260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3606640Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3607003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3607379Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3607751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.3608129Z     key_states = self.k(current_states)
2025-09-07T08:12:53.3608258Z 
2025-09-07T08:12:53.3608361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3608716Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3609029Z     return mod(**inputs)
2025-09-07T08:12:53.3609387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3609758Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3610118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3610546Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3610958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3611318Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3611680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3612053Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3612411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3612783Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3613147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3613565Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3613748Z 
2025-09-07T08:12:53.3613849Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3614244Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3614572Z     return mod(**inputs)
2025-09-07T08:12:53.3614919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3615331Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3615696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3616061Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3616403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3616761Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3617137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3617510Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3617942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3618329Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3618696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3619113Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3619290Z 
2025-09-07T08:12:53.3619392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3619746Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3620066Z     return mod(**inputs)
2025-09-07T08:12:53.3620410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3620770Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3621133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3621500Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3621844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3622202Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3622559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3622927Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3623294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3623668Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3624036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3624486Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3624710Z 
2025-09-07T08:12:53.3624809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3625168Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3625485Z     return mod(**inputs)
2025-09-07T08:12:53.3625840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3626207Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3626566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3626931Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3627267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3627618Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3628033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3628441Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3628836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3629222Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3629579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3630027Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3630241Z 
2025-09-07T08:12:53.3630337Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3630690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3631004Z     return mod(**inputs)
2025-09-07T08:12:53.3631340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3631707Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3632067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3632433Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3632759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3633119Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3633481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3633864Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3634235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3634622Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3634988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.3635357Z     value_states = self.v(current_states)
2025-09-07T08:12:53.3635492Z 
2025-09-07T08:12:53.3635594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3635939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3636244Z     return mod(**inputs)
2025-09-07T08:12:53.3636576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3636946Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3637297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3637659Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3638030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3638434Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3638797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3639166Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3639528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3639897Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3640261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3640662Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3640821Z 
2025-09-07T08:12:53.3640916Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3641262Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3641626Z     return mod(**inputs)
2025-09-07T08:12:53.3641963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3642356Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3642743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3643131Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3643478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3643842Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3644198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3644567Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3644936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3645309Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3645667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3646067Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3646225Z 
2025-09-07T08:12:53.3646339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3646681Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3646997Z     return mod(**inputs)
2025-09-07T08:12:53.3647318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3647689Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3648045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3648411Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3648748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3649094Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3649459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3649830Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3650190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3650558Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3650917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3651303Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3651502Z 
2025-09-07T08:12:53.3651593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3651974Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3652294Z     return mod(**inputs)
2025-09-07T08:12:53.3652637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3653001Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3653361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3653730Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3654053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3654413Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3654769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3655182Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3655544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3655911Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3656303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.3656705Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.3656857Z 
2025-09-07T08:12:53.3656954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3657302Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3657645Z     return mod(**inputs)
2025-09-07T08:12:53.3657978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3658350Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3658710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3659070Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3659402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3659755Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3660112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3660482Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3660842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3661207Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3661573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.3661941Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.3662069Z 
2025-09-07T08:12:53.3662172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3662508Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3662832Z     return mod(**inputs)
2025-09-07T08:12:53.3663165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.3663544Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.3663892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3664261Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3664590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3665033Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3665440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.3665819Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.3666189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.3666571Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.3666935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.3667302Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.3667442Z 
2025-09-07T08:12:53.3667542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3667898Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3668229Z     return mod(**inputs)
2025-09-07T08:12:53.3668607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3668983Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3669343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3669742Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3670087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3670447Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3670800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3671189Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3671552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3671925Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3672283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.3672651Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.3672788Z 
2025-09-07T08:12:53.3672887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3673233Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3673547Z     return mod(**inputs)
2025-09-07T08:12:53.3673879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3674244Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3674601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3674964Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3675298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3675654Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3676018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3676393Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3676755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3677120Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3677486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.3677851Z     key_states = self.k(current_states)
2025-09-07T08:12:53.3677979Z 
2025-09-07T08:12:53.3678082Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3678427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3678815Z     return mod(**inputs)
2025-09-07T08:12:53.3679152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3679520Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3679875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3680234Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3680566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3680920Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3681286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3681659Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3682014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3682431Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3682794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3684293Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3684485Z 
2025-09-07T08:12:53.3684592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3684943Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3685261Z     return mod(**inputs)
2025-09-07T08:12:53.3685599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3685969Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3686315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3686683Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3687018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3687373Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3687733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3688094Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3688453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3688825Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3689186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3689588Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3689772Z 
2025-09-07T08:12:53.3689868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3690213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3690528Z     return mod(**inputs)
2025-09-07T08:12:53.3690861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3691221Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3691574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3691939Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3692272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3692624Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3692976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3693473Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3693854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3694230Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3694598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3695010Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3695193Z 
2025-09-07T08:12:53.3695287Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3695627Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3695938Z     return mod(**inputs)
2025-09-07T08:12:53.3696262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3696632Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3697022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3697394Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3697810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3698166Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3698522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3698889Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3699264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3699628Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3699989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3700444Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3700651Z 
2025-09-07T08:12:53.3700755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3701102Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3701410Z     return mod(**inputs)
2025-09-07T08:12:53.3701742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3702106Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3702456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3702818Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3703320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3703683Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3704052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3704419Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3704775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3705143Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3705504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3705943Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3706149Z 
2025-09-07T08:12:53.3706250Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3706585Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3706980Z     return mod(**inputs)
2025-09-07T08:12:53.3707362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3707726Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3708074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3708437Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3708767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3709122Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3709477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3709842Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3710199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3710673Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3711037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.3711398Z     value_states = self.v(current_states)
2025-09-07T08:12:53.3711571Z 
2025-09-07T08:12:53.3711669Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3712014Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3712328Z     return mod(**inputs)
2025-09-07T08:12:53.3712662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3713024Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3713376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3713739Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3714073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3714430Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3714782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3715154Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3715511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3715877Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3716235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3716626Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3716792Z 
2025-09-07T08:12:53.3716883Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3717229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3717544Z     return mod(**inputs)
2025-09-07T08:12:53.3717882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3718265Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3718634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3719003Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3719352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3719709Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3720080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3720500Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3720911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3721281Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3721650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3722049Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3722209Z 
2025-09-07T08:12:53.3722317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3722671Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3722987Z     return mod(**inputs)
2025-09-07T08:12:53.3723329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3723699Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3724090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3724460Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3724798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3725187Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3725559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3725932Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3726287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3726655Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3727018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3727414Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3727570Z 
2025-09-07T08:12:53.3727677Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3728017Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3728336Z     return mod(**inputs)
2025-09-07T08:12:53.3728671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3729039Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3729390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3729749Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3730086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3730443Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3730806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3731171Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3731537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3731912Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3732274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.3732663Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.3732818Z 
2025-09-07T08:12:53.3732913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3733260Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3733573Z     return mod(**inputs)
2025-09-07T08:12:53.3733913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3734358Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3734727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3735099Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3735447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3735813Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3736184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3736568Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3736956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3737332Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3737800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.3738187Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.3738319Z 
2025-09-07T08:12:53.3738397Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3738667Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3739025Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3739339Z     return mod(**inputs)
2025-09-07T08:12:53.3739687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3740061Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3740420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3740785Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3741128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3741491Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3741850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3742237Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3742612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.3743002Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3743384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3743753Z     return self.weight * hidden_states
2025-09-07T08:12:53.3743880Z 
2025-09-07T08:12:53.3743988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3744336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3744664Z     return mod(**inputs)
2025-09-07T08:12:53.3745008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3745380Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3745738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3746106Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3746443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3746801Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3747168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3747542Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3747981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3748431Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3748848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.3749220Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.3749350Z 
2025-09-07T08:12:53.3749450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3749802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3750123Z     return mod(**inputs)
2025-09-07T08:12:53.3750462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3750832Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3751186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3751595Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3751943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3752334Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3752695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3753081Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3753458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3753868Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3754265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.3754631Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.3754768Z 
2025-09-07T08:12:53.3754864Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3755213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3755528Z     return mod(**inputs)
2025-09-07T08:12:53.3755857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3756223Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3756576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3756938Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3757271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3757619Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3757982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3758363Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3758740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3759143Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3759533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.3759905Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.3760039Z 
2025-09-07T08:12:53.3760116Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3760347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3760688Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3761000Z     return mod(**inputs)
2025-09-07T08:12:53.3761379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3761788Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3762140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3762502Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3762839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3763192Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3763553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3763917Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3764281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.3764696Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3765122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3765494Z     return self.weight * hidden_states
2025-09-07T08:12:53.3765621Z 
2025-09-07T08:12:53.3765748Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3766097Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3766414Z     return mod(**inputs)
2025-09-07T08:12:53.3766750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3767118Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3767466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3767832Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3768171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3768529Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3768885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3769259Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3769621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3769995Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3770358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.3770720Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.3770854Z 
2025-09-07T08:12:53.3770948Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3771290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3771611Z     return mod(**inputs)
2025-09-07T08:12:53.3771948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3772307Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3772664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3773028Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3773363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3773711Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3774068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3774439Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3774844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3775253Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3775609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.3775988Z     key_states = self.k(current_states)
2025-09-07T08:12:53.3776124Z 
2025-09-07T08:12:53.3776221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3776582Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3776888Z     return mod(**inputs)
2025-09-07T08:12:53.3777238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3777670Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3778032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3778404Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3778764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3779121Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3779510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3779890Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3780254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3780616Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3780981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3781397Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3781583Z 
2025-09-07T08:12:53.3781685Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3782028Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3782344Z     return mod(**inputs)
2025-09-07T08:12:53.3782679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3783050Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3783403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3783758Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3784090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3784443Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3784802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3785174Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3785530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3785900Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3786263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3786674Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3786849Z 
2025-09-07T08:12:53.3786942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3787286Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3787604Z     return mod(**inputs)
2025-09-07T08:12:53.3787935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3788357Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3788741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3789106Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3789435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3789794Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3790148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3790521Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3790878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3791243Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3791605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3792068Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3792251Z 
2025-09-07T08:12:53.3792343Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3792731Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3793055Z     return mod(**inputs)
2025-09-07T08:12:53.3793392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3793754Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3794111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3794474Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3794805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3795154Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3795517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3795888Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3796253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3796621Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3796977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3797429Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3797640Z 
2025-09-07T08:12:53.3797734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3798075Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3798391Z     return mod(**inputs)
2025-09-07T08:12:53.3798721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3799087Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3799444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3799809Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3800135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3800490Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3800846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3801216Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3801575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3802023Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3802382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3802818Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3803019Z 
2025-09-07T08:12:53.3803275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3803620Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3803928Z     return mod(**inputs)
2025-09-07T08:12:53.3804264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3804630Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3804987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3805349Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3805759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3806124Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3806542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3806927Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3807290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3807662Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3808026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.3808392Z     value_states = self.v(current_states)
2025-09-07T08:12:53.3808523Z 
2025-09-07T08:12:53.3808627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3808975Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3809293Z     return mod(**inputs)
2025-09-07T08:12:53.3809625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3809996Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3810343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3810703Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3811034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3811386Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3811746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3812115Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3812483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3812848Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3813212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3813606Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3813775Z 
2025-09-07T08:12:53.3813870Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3814213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3814526Z     return mod(**inputs)
2025-09-07T08:12:53.3814855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3815214Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3815626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3816044Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3816386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3816748Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3817104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3817476Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3817898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3818270Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3818622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3819021Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3819184Z 
2025-09-07T08:12:53.3819315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3819659Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3820009Z     return mod(**inputs)
2025-09-07T08:12:53.3820335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3820704Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3821058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3821421Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3821748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3822101Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3822468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3822841Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3823200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3823565Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3823933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3824324Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3824478Z 
2025-09-07T08:12:53.3824583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3824927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3825232Z     return mod(**inputs)
2025-09-07T08:12:53.3825565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3825961Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3826311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3826668Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3826999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3827353Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3827714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3828080Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3828438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3828827Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3829248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.3829688Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.3829848Z 
2025-09-07T08:12:53.3829967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3830319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3830646Z     return mod(**inputs)
2025-09-07T08:12:53.3830992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3831364Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3831720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3832088Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3843629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3844158Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3844565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3844999Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3845378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3845796Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3846172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.3846574Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.3846702Z 
2025-09-07T08:12:53.3846813Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3847179Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3847504Z     return mod(**inputs)
2025-09-07T08:12:53.3847874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3848245Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3848603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3848959Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3849327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3849679Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3850037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3850431Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3850808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.3851200Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3851574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3851931Z     return self.weight * hidden_states
2025-09-07T08:12:53.3852082Z 
2025-09-07T08:12:53.3852180Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3852526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3852855Z     return mod(**inputs)
2025-09-07T08:12:53.3853192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3853571Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3853930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3854343Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3854732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3855083Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3855455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3855838Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3856235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3856657Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3857050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.3857432Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.3857650Z 
2025-09-07T08:12:53.3857750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3858163Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3858484Z     return mod(**inputs)
2025-09-07T08:12:53.3858856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3859231Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3859589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3859951Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3860292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3860646Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3861012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3861396Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3861773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3862184Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3862583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.3862957Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.3863087Z 
2025-09-07T08:12:53.3863192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3863537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3863862Z     return mod(**inputs)
2025-09-07T08:12:53.3864214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3864583Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3864947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3865317Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3865668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3866015Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3866377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3866754Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3867120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3867531Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3867920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.3868372Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.3868502Z 
2025-09-07T08:12:53.3868577Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3868812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3869155Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3869477Z     return mod(**inputs)
2025-09-07T08:12:53.3869806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3870167Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3870537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3870909Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3871234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3871580Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3872171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3872541Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3872937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.3873334Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3873714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3874084Z     return self.weight * hidden_states
2025-09-07T08:12:53.3874210Z 
2025-09-07T08:12:53.3874304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3874651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3874961Z     return mod(**inputs)
2025-09-07T08:12:53.3875292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3875659Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3876009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3876374Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3876697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3877068Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3877413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3877771Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3878168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3878532Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3878898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.3879257Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.3879384Z 
2025-09-07T08:12:53.3879479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3879812Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3880116Z     return mod(**inputs)
2025-09-07T08:12:53.3880438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3880794Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3881143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3881548Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3881918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3882262Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3882622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3882989Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3883334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3883706Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3884056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.3884411Z     key_states = self.k(current_states)
2025-09-07T08:12:53.3884541Z 
2025-09-07T08:12:53.3884634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3885007Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3885325Z     return mod(**inputs)
2025-09-07T08:12:53.3885648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3886039Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3886393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3886754Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3887085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3887447Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3887808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3888183Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3888554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3888922Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3889283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3889690Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3889867Z 
2025-09-07T08:12:53.3889963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3890314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3890632Z     return mod(**inputs)
2025-09-07T08:12:53.3890969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3891326Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3891681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3892041Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3892369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3892715Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3893075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3893451Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3893821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3894180Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3894531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3894991Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3895197Z 
2025-09-07T08:12:53.3895294Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3895634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3895941Z     return mod(**inputs)
2025-09-07T08:12:53.3896269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3896625Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3896969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3897326Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3897729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3898079Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3898484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3898857Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3899221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3899616Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3899969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3900375Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3900552Z 
2025-09-07T08:12:53.3900646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3900984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3901297Z     return mod(**inputs)
2025-09-07T08:12:53.3901623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3901986Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3902327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3902681Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3903016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3903575Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3903939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3904305Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3904663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3905021Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3905384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3905821Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3906024Z 
2025-09-07T08:12:53.3906125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3906471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3906777Z     return mod(**inputs)
2025-09-07T08:12:53.3907105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3907465Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3907810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3908162Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3908590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3909004Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3909354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3909716Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3910067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3910435Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3910789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3911225Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3911426Z 
2025-09-07T08:12:53.3911525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3911869Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3912247Z     return mod(**inputs)
2025-09-07T08:12:53.3912579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3912931Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3913323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3913686Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3914021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3914371Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3914723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3915080Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3915433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3915791Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3916146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.3916509Z     value_states = self.v(current_states)
2025-09-07T08:12:53.3916637Z 
2025-09-07T08:12:53.3916728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3917062Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3917364Z     return mod(**inputs)
2025-09-07T08:12:53.3917685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3918037Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3918376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3918735Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3919059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3919407Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3919761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3920122Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3920475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3920838Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3921194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3921587Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3921791Z 
2025-09-07T08:12:53.3921882Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3922302Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3922606Z     return mod(**inputs)
2025-09-07T08:12:53.3922926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3923278Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3923621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3923975Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3924303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3924642Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3924994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3925390Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3925740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3926102Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3926487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3926878Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3927030Z 
2025-09-07T08:12:53.3927126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3927459Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3927761Z     return mod(**inputs)
2025-09-07T08:12:53.3928080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3928436Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3928777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3929127Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3929448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3929791Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3930146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3930510Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3930858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3931215Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3931572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3931962Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3932109Z 
2025-09-07T08:12:53.3932201Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3932527Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3932830Z     return mod(**inputs)
2025-09-07T08:12:53.3933149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3933506Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3933851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3934202Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3934526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3934917Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3935348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3935705Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3936055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3936410Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3936758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.3937137Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.3937285Z 
2025-09-07T08:12:53.3937374Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3937753Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3938057Z     return mod(**inputs)
2025-09-07T08:12:53.3938414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3938779Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3939148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3939509Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3939840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3940180Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3940529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3940890Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3941240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3941603Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3941958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.3942314Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.3942438Z 
2025-09-07T08:12:53.3942512Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3942727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3943065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3943377Z     return mod(**inputs)
2025-09-07T08:12:53.3943711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3944072Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3944415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3944772Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3945099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3945448Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3945800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3946172Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3946535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.3946913Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3947300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3947667Z     return self.weight * hidden_states
2025-09-07T08:12:53.3947794Z 
2025-09-07T08:12:53.3947956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3948338Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3948653Z     return mod(**inputs)
2025-09-07T08:12:53.3948990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3949352Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3949704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3950071Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3950408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3950769Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3951126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3951497Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3951901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3952306Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3952733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.3953101Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.3953230Z 
2025-09-07T08:12:53.3953328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3953674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3953981Z     return mod(**inputs)
2025-09-07T08:12:53.3954308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3954667Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3955013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3955370Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3955698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3956042Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3956389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3956754Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3957112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3957507Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3957908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.3958277Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.3958414Z 
2025-09-07T08:12:53.3958510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3958852Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3959169Z     return mod(**inputs)
2025-09-07T08:12:53.3959501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3959860Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3960223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3960585Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3960916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3961260Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3961684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3962093Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3962465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.3962861Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.3963249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.3963613Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.3963617Z 
2025-09-07T08:12:53.3963695Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3963788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3963977Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3964046Z     return mod(**inputs)
2025-09-07T08:12:53.3964314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3964388Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3964650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3964713Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3964936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3965006Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3965242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3965315Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3965542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.3965651Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3965878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3965952Z     return self.weight * hidden_states
2025-09-07T08:12:53.3965957Z 
2025-09-07T08:12:53.3966048Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3966241Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3966299Z     return mod(**inputs)
2025-09-07T08:12:53.3966530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3966600Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3966832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3966902Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3967120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3967189Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3967424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3967496Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3967728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3967804Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3968036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.3968108Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.3968112Z 
2025-09-07T08:12:53.3968201Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3968429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3968517Z     return mod(**inputs)
2025-09-07T08:12:53.3968756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3968822Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3969054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3969123Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3969340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3969416Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3969644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3969717Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3969987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3970061Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3970325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.3970396Z     key_states = self.k(current_states)
2025-09-07T08:12:53.3970400Z 
2025-09-07T08:12:53.3970493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3970679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3970739Z     return mod(**inputs)
2025-09-07T08:12:53.3970978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3971041Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3971281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3971346Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3971559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3971633Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3971859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3971937Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3972166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3972237Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3972471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3972589Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3972595Z 
2025-09-07T08:12:53.3972694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3972877Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3972940Z     return mod(**inputs)
2025-09-07T08:12:53.3973171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3973236Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3973472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3973534Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3973751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3973818Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3974099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3974212Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3974443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3974524Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3974751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3974866Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3974876Z 
2025-09-07T08:12:53.3974973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3975165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3975229Z     return mod(**inputs)
2025-09-07T08:12:53.3975464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3975568Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3975805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3975872Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3976125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3976197Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3976435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3976506Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3976732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3976813Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3977050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.3977171Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.3977175Z 
2025-09-07T08:12:53.3977264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3977456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3977514Z     return mod(**inputs)
2025-09-07T08:12:53.3977838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3977909Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3978141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3978209Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3978426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3978501Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3978734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3978806Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3979039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3979110Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3979339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3979494Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3979497Z 
2025-09-07T08:12:53.3979588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3979780Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3979904Z     return mod(**inputs)
2025-09-07T08:12:53.3980145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3980208Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3980442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3980510Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3980724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3980797Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3981024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3981095Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3981326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3981430Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3981669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.3981849Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.3981853Z 
2025-09-07T08:12:53.3981951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3982136Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3982194Z     return mod(**inputs)
2025-09-07T08:12:53.3982431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3982493Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3982733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3982800Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3983012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3983086Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3983317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3983396Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3983623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3983694Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3983923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.3983990Z     value_states = self.v(current_states)
2025-09-07T08:12:53.3983995Z 
2025-09-07T08:12:53.3984084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3984267Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3984323Z     return mod(**inputs)
2025-09-07T08:12:53.3984553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3984614Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3984844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3984902Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3985117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3985180Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3985405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3985538Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3985764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3985834Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3986061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3986160Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3986166Z 
2025-09-07T08:12:53.3986254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3986436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3986493Z     return mod(**inputs)
2025-09-07T08:12:53.3986722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3986787Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3987059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3987119Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3987374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3987439Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3987667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3987734Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3987959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3988028Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3988252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3988350Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3988353Z 
2025-09-07T08:12:53.3988439Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3988623Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3988678Z     return mod(**inputs)
2025-09-07T08:12:53.3988905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3988968Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3989197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3989257Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3989467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3989533Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3989763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3989829Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3990058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3990125Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3990349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.3990439Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.3990443Z 
2025-09-07T08:12:53.3990529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3990710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3990800Z     return mod(**inputs)
2025-09-07T08:12:53.3991032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3991124Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3991354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3991418Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3991629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3991695Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3991920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3991987Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3992216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3992285Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3992542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.3992633Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.3992636Z 
2025-09-07T08:12:53.3992753Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3992936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3992990Z     return mod(**inputs)
2025-09-07T08:12:53.3993219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3993279Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3993509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3993568Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3993782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3993851Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3994076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3994146Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3994370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.3994438Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.3994665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.3994731Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.3994734Z 
2025-09-07T08:12:53.3994825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3995009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3995070Z     return mod(**inputs)
2025-09-07T08:12:53.3995303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3995368Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3995601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3995662Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3995877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3995946Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3996172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.3996244Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.3996508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:12:53.3996672Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:12:53.3996675Z 
2025-09-07T08:12:53.3996746Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.3996839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3997027Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3997082Z     return mod(**inputs)
2025-09-07T08:12:53.3997315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3997377Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3997610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3997671Z     layer_outputs = layer_module(
2025-09-07T08:12:53.3997918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.3997989Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.3998246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.3998329Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.3998555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.3998641Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.3998871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.3998936Z     return self.weight * hidden_states
2025-09-07T08:12:53.3998939Z 
2025-09-07T08:12:53.3999033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.3999221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.3999279Z     return mod(**inputs)
2025-09-07T08:12:53.3999512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.3999577Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.3999811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.3999871Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4000083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4000149Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4000374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4000454Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4000685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4000795Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4001023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4001094Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4001097Z 
2025-09-07T08:12:53.4001191Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4001375Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4001434Z     return mod(**inputs)
2025-09-07T08:12:53.4001661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4001726Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4001988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4002081Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4002298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4002367Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4002599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4002676Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4002901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4003006Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4003468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4003544Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4003549Z 
2025-09-07T08:12:53.4003718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4003910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4003971Z     return mod(**inputs)
2025-09-07T08:12:53.4004261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4004330Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4004562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4004626Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4004842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4004910Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4005149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4005233Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4005462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4005568Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4005796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4005869Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4005872Z 
2025-09-07T08:12:53.4005944Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4006038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4006225Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4006281Z     return mod(**inputs)
2025-09-07T08:12:53.4006522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4006589Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4006826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4006891Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4007113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4007186Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4007415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4007494Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4007721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4007872Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4008155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4008227Z     return self.weight * hidden_states
2025-09-07T08:12:53.4008230Z 
2025-09-07T08:12:53.4008335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4008526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4008594Z     return mod(**inputs)
2025-09-07T08:12:53.4008830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4008897Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4009136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4009202Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4009458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4009531Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4009769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4009874Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4010107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4010189Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4010415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4010493Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4010496Z 
2025-09-07T08:12:53.4010588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4010780Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4010853Z     return mod(**inputs)
2025-09-07T08:12:53.4011088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4011158Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4011388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4011456Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4011672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4011740Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4011975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4012050Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4012287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4012366Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4012595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4012671Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4012675Z 
2025-09-07T08:12:53.4012762Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4012954Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4013011Z     return mod(**inputs)
2025-09-07T08:12:53.4013243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4013308Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4013540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4013669Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4013885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4013956Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4014186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4014258Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4014491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4014561Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4014790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4014914Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4014919Z 
2025-09-07T08:12:53.4015010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4015228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4015284Z     return mod(**inputs)
2025-09-07T08:12:53.4015553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4015616Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4015846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4015932Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4016156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4016228Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4016454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4016532Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4016757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4016827Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4017076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4017186Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4017190Z 
2025-09-07T08:12:53.4017283Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4017471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4017536Z     return mod(**inputs)
2025-09-07T08:12:53.4017830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4017896Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4018143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4018203Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4018417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4018486Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4018715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4018795Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4019020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4019089Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4019312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4019487Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4019490Z 
2025-09-07T08:12:53.4019583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4019769Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4019828Z     return mod(**inputs)
2025-09-07T08:12:53.4020057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4020124Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4020363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4020426Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4020641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4020709Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4020971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4021038Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4021301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4021375Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4021601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4021765Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4021768Z 
2025-09-07T08:12:53.4021854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4022039Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4022097Z     return mod(**inputs)
2025-09-07T08:12:53.4022332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4022402Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4022630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4022689Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4022907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4022972Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4023211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4023279Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4023520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4023591Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4023820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4023963Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4023966Z 
2025-09-07T08:12:53.4024055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4024241Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4024296Z     return mod(**inputs)
2025-09-07T08:12:53.4024527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4024601Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4024833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4024934Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4025195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4025272Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4025504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4025583Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4025822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4025906Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4026138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4026208Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4026211Z 
2025-09-07T08:12:53.4026304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4026526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4026584Z     return mod(**inputs)
2025-09-07T08:12:53.4026853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4026919Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4027151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4027232Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4027446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4027531Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4027758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4027833Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4028061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4028133Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4028365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4028466Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4028470Z 
2025-09-07T08:12:53.4028564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4028747Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4028803Z     return mod(**inputs)
2025-09-07T08:12:53.4029051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4029114Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4029355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4029429Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4029647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4029722Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4029952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4030028Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4030263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4030334Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4030563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4030688Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4030720Z 
2025-09-07T08:12:53.4030815Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4031006Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4031067Z     return mod(**inputs)
2025-09-07T08:12:53.4031297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4031359Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4031590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4031651Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4031882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4031946Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4032205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4032283Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4032543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4032620Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4032845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4032938Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4033199Z 
2025-09-07T08:12:53.4033306Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4033503Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4033568Z     return mod(**inputs)
2025-09-07T08:12:53.4033805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4033898Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4034128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4034192Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4034402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4034466Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4034709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4034776Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4035008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4035082Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4035316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4035415Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4035419Z 
2025-09-07T08:12:53.4035509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4035700Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4035758Z     return mod(**inputs)
2025-09-07T08:12:53.4035991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4036059Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4036289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4036366Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4036580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4036710Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4036940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4037011Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4037251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4037320Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4037553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4037620Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4037624Z 
2025-09-07T08:12:53.4037696Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4037792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4037978Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4038071Z     return mod(**inputs)
2025-09-07T08:12:53.4038307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4038403Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4038651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4038712Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4038933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4038999Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4039236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4039319Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4039549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4039640Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4039868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4039947Z     return self.weight * hidden_states
2025-09-07T08:12:53.4039951Z 
2025-09-07T08:12:53.4040039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4040223Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4040284Z     return mod(**inputs)
2025-09-07T08:12:53.4040513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4040579Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4040818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4040883Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4041097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4041165Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4041396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4041472Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4041708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4041815Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4042042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4042182Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4042213Z 
2025-09-07T08:12:53.4042304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4042492Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4042549Z     return mod(**inputs)
2025-09-07T08:12:53.4042781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4042846Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4043077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4043145Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4043356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4043426Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4043652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4043765Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4043996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4044123Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4044367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4044438Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4044442Z 
2025-09-07T08:12:53.4044529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4044717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4044771Z     return mod(**inputs)
2025-09-07T08:12:53.4045005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4045071Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4045302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4045366Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4045582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4045654Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4045883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4045964Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4046191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4046290Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4046524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4046592Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4046595Z 
2025-09-07T08:12:53.4046670Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4046758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4046940Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4047000Z     return mod(**inputs)
2025-09-07T08:12:53.4047236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4047306Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4047536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4047598Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4047864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4047959Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4048194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4048264Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4048499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4048596Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4048826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4048896Z     return self.weight * hidden_states
2025-09-07T08:12:53.4048899Z 
2025-09-07T08:12:53.4048985Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4049176Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4049263Z     return mod(**inputs)
2025-09-07T08:12:53.4049499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4049567Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4049829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4049898Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4050111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4050181Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4050409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4050478Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4050712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4050783Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4051015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4051083Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4051086Z 
2025-09-07T08:12:53.4051173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4051358Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4051416Z     return mod(**inputs)
2025-09-07T08:12:53.4051651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4051711Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4051941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4052011Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4052220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4052289Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4052517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4052586Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4052810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4052878Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4053107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4053178Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4053214Z 
2025-09-07T08:12:53.4053303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4053517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4053573Z     return mod(**inputs)
2025-09-07T08:12:53.4053808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4053869Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4054100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4054162Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4054375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4054443Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4054671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4054747Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4055005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4055080Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4055340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4055462Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4055465Z 
2025-09-07T08:12:53.4055562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4055747Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4055804Z     return mod(**inputs)
2025-09-07T08:12:53.4056035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4056101Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4056336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4056398Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4056619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4056684Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4056910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4056984Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4057210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4057282Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4057509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4057683Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4057687Z 
2025-09-07T08:12:53.4057776Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4057960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4058020Z     return mod(**inputs)
2025-09-07T08:12:53.4058251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4058315Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4058546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4058607Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4058822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4058926Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4059190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4059261Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4059491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4059560Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4059785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4059899Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4059903Z 
2025-09-07T08:12:53.4059988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4060181Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4060238Z     return mod(**inputs)
2025-09-07T08:12:53.4060499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4060566Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4060828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4060897Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4061109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4061176Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4061408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4061478Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4061712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4061783Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4062017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4062162Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4062167Z 
2025-09-07T08:12:53.4062258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4062443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4062498Z     return mod(**inputs)
2025-09-07T08:12:53.4062735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4062799Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4063026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4063096Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4063310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4063377Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4063605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4063678Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4063906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4063973Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4064202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4064336Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4064340Z 
2025-09-07T08:12:53.4064469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4064682Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4064739Z     return mod(**inputs)
2025-09-07T08:12:53.4064971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4065033Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4065276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4065335Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4065546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4065613Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4065838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4065909Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4066168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4066242Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4066516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4066586Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4066589Z 
2025-09-07T08:12:53.4066682Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4066863Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4066925Z     return mod(**inputs)
2025-09-07T08:12:53.4067153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4067216Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4067459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4067525Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4067746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4067815Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4068046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4068122Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4068351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4068431Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4068659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4068767Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4068771Z 
2025-09-07T08:12:53.4068863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4069047Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4069113Z     return mod(**inputs)
2025-09-07T08:12:53.4069344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4069414Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4069645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4069710Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4069928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4069995Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4070527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4070627Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4070865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4070938Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4071165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4071269Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4071273Z 
2025-09-07T08:12:53.4071364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4071559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4071617Z     return mod(**inputs)
2025-09-07T08:12:53.4071854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4071959Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4072194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4072295Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4072511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4072580Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4072815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4072886Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4073121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4073193Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4073430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4073526Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4073530Z 
2025-09-07T08:12:53.4073621Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4073814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4073873Z     return mod(**inputs)
2025-09-07T08:12:53.4074115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4074178Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4074409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4074480Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4074692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4074773Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4075001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4075075Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4075309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4075381Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4075617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4075708Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4075711Z 
2025-09-07T08:12:53.4075806Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4075987Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4076104Z     return mod(**inputs)
2025-09-07T08:12:53.4076345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4076410Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4076647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4076710Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4076932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4077095Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4077355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4077671Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4077932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4078068Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4078365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4078493Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4078496Z 
2025-09-07T08:12:53.4078667Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4087096Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4087172Z     return mod(**inputs)
2025-09-07T08:12:53.4087447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4087519Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4087770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4087849Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4088087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4088169Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4088415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4088496Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4088725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:12:53.4088849Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:12:53.4088853Z 
2025-09-07T08:12:53.4088934Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4089035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4089242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4089306Z     return mod(**inputs)
2025-09-07T08:12:53.4089559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4089636Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4089890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4089961Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4090186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4090262Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4090505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4090589Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4090823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4091052Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4091284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4091359Z     return self.weight * hidden_states
2025-09-07T08:12:53.4091363Z 
2025-09-07T08:12:53.4091463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4091675Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4091738Z     return mod(**inputs)
2025-09-07T08:12:53.4091983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4092051Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4092283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4092356Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4092607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4092688Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4092954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4093038Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4093271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4093384Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4093618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4093690Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4093695Z 
2025-09-07T08:12:53.4093795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4093989Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4094046Z     return mod(**inputs)
2025-09-07T08:12:53.4094285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4094351Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4094587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4094649Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4094863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4094937Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4095166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4095252Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4095477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4095584Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4095812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4095886Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4095889Z 
2025-09-07T08:12:53.4095986Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4096176Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4096237Z     return mod(**inputs)
2025-09-07T08:12:53.4096468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4096568Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4096842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4096907Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4097129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4097200Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4097428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4097514Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4097812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4097924Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4098156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4098269Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4098273Z 
2025-09-07T08:12:53.4098347Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4098440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4098662Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4098721Z     return mod(**inputs)
2025-09-07T08:12:53.4098956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:12:53.4099020Z     encoder_outputs = self.encoder(
2025-09-07T08:12:53.4099249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward
2025-09-07T08:12:53.4099346Z     hidden_states = self.final_layer_norm(hidden_states)
2025-09-07T08:12:53.4099575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4099650Z     return self.weight * hidden_states
2025-09-07T08:12:53.4099654Z 
2025-09-07T08:12:53.4099746Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4099934Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4099991Z     return mod(**inputs)
2025-09-07T08:12:53.4100218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4100282Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4100511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4100576Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4100788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4100859Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4101091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4101161Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4101392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4101465Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4101690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4101760Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4101764Z 
2025-09-07T08:12:53.4101852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4102035Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4102092Z     return mod(**inputs)
2025-09-07T08:12:53.4102360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4102450Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4102683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4102746Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4102958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4103026Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4103440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4103513Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4103743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4103816Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4104107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4104230Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4104233Z 
2025-09-07T08:12:53.4104364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4104553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4104612Z     return mod(**inputs)
2025-09-07T08:12:53.4104851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4104914Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4105147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4105206Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4105422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4105496Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4105726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4105797Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4106034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4106109Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4106335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4106446Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4106450Z 
2025-09-07T08:12:53.4106543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4106729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4106789Z     return mod(**inputs)
2025-09-07T08:12:53.4107026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4107090Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4107327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4107387Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4107600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4107665Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4107890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4108008Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4108292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4108367Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4108595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4108704Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4108708Z 
2025-09-07T08:12:53.4108797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4108979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4109037Z     return mod(**inputs)
2025-09-07T08:12:53.4109269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4109332Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4109637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4109699Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4109913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4110023Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4110259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4110329Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4110556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4110632Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4110855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4111009Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4111015Z 
2025-09-07T08:12:53.4111107Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4111297Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4111358Z     return mod(**inputs)
2025-09-07T08:12:53.4111595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4111663Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4111893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4111962Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4112174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4112241Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4112479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4112555Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4112794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4112866Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4113093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4113173Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4113177Z 
2025-09-07T08:12:53.4113271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4113464Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4113525Z     return mod(**inputs)
2025-09-07T08:12:53.4113758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4113893Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4114124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4114198Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4114413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4114488Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4114716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4114785Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4115015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4115085Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4115346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4115449Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4115453Z 
2025-09-07T08:12:53.4115543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4115758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4115821Z     return mod(**inputs)
2025-09-07T08:12:53.4116059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4116121Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4116377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4116441Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4116667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4116741Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4116974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4117047Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4117273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4117348Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4117579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4117675Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4117679Z 
2025-09-07T08:12:53.4117774Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4117967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4118031Z     return mod(**inputs)
2025-09-07T08:12:53.4118271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4118335Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4118570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4118633Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4118850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4118917Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4119141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4119217Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4119485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4119595Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4119821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4119915Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4119919Z 
2025-09-07T08:12:53.4120015Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4120199Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4120262Z     return mod(**inputs)
2025-09-07T08:12:53.4120493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4120557Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4120790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4120854Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4121102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4121172Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4121437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4121507Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4121736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4121815Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4122043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4122141Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4122146Z 
2025-09-07T08:12:53.4122236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4122426Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4122489Z     return mod(**inputs)
2025-09-07T08:12:53.4122719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4122786Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4123014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4123080Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4123290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4123356Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4123588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4123662Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4123893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4123967Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4124196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4124273Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4124277Z 
2025-09-07T08:12:53.4124353Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4124449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4124633Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4124690Z     return mod(**inputs)
2025-09-07T08:12:53.4124924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4125046Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4125283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4125343Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4125558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4125625Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4125850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4125939Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4126165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4126258Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4126519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4126590Z     return self.weight * hidden_states
2025-09-07T08:12:53.4126593Z 
2025-09-07T08:12:53.4126688Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4126903Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4126968Z     return mod(**inputs)
2025-09-07T08:12:53.4127200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4127265Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4127499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4127560Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4127779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4127847Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4128080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4128159Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4128388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4128501Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4128726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4128808Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4128813Z 
2025-09-07T08:12:53.4128903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4129095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4129153Z     return mod(**inputs)
2025-09-07T08:12:53.4129387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4129448Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4129683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4129744Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4129956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4130028Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4130252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4130332Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4130556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4130720Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4130953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4131028Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4131031Z 
2025-09-07T08:12:53.4131126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4131308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4131372Z     return mod(**inputs)
2025-09-07T08:12:53.4131604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4131667Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4131900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4131964Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4132228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4132296Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4132558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4132643Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4132872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4132977Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4133204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4133272Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4133282Z 
2025-09-07T08:12:53.4133370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4133557Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4133615Z     return mod(**inputs)
2025-09-07T08:12:53.4133845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4133913Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4134142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4134201Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4134413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4134480Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4134710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4134783Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4135011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4135112Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4135341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4135412Z     return self.weight * hidden_states
2025-09-07T08:12:53.4135415Z 
2025-09-07T08:12:53.4135504Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4135690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4135747Z     return mod(**inputs)
2025-09-07T08:12:53.4135978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4136077Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4136313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4136417Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4136632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4136699Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4136933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4137002Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4137234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4137306Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4137532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4137669Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4137703Z 
2025-09-07T08:12:53.4137795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4137984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4138074Z     return mod(**inputs)
2025-09-07T08:12:53.4138310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4138378Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4138608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4138676Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4138888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4138957Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4139187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4139259Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4139495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4139564Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4139796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4139866Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4139869Z 
2025-09-07T08:12:53.4139959Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4140145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4140201Z     return mod(**inputs)
2025-09-07T08:12:53.4140434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4140500Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4140737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4140798Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4141016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4141085Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4141310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4141380Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4141605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4141674Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4141940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4142087Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4142091Z 
2025-09-07T08:12:53.4142184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4142368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4142423Z     return mod(**inputs)
2025-09-07T08:12:53.4142659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4142720Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4142955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4143014Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4143227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4143334Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4143566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4143671Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4143901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4143979Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4144206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4144320Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4144323Z 
2025-09-07T08:12:53.4144419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4144605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4144670Z     return mod(**inputs)
2025-09-07T08:12:53.4144903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4144966Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4145200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4145262Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4145481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4145548Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4145780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4145850Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4146082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4146160Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4146384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4146499Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4146502Z 
2025-09-07T08:12:53.4146593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4146773Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4146834Z     return mod(**inputs)
2025-09-07T08:12:53.4147061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4147129Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4147359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4147494Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4147707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4147773Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4148005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4148075Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4148307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4148378Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4148605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4148756Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4148761Z 
2025-09-07T08:12:53.4148882Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4149069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4149126Z     return mod(**inputs)
2025-09-07T08:12:53.4149550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4149621Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4149849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4149915Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4150124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4150195Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4150422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4150496Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4150725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4150796Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4151026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4151096Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4151099Z 
2025-09-07T08:12:53.4151187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4151377Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4151434Z     return mod(**inputs)
2025-09-07T08:12:53.4151668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4151731Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4151963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4152024Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4152235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4152309Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4152536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4152607Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4152836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4152907Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4153139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4153314Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4153318Z 
2025-09-07T08:12:53.4153412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4153594Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4153652Z     return mod(**inputs)
2025-09-07T08:12:53.4153888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4153949Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4154180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4154241Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4154457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4154527Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4154788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4154869Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4155127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4155200Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4155428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4155519Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4155522Z 
2025-09-07T08:12:53.4155616Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4155803Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4155866Z     return mod(**inputs)
2025-09-07T08:12:53.4156096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4156159Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4156390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4156451Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4156669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4156736Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4156969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4157038Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4157263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4157341Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4157570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4157663Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4157667Z 
2025-09-07T08:12:53.4157755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4157937Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4158001Z     return mod(**inputs)
2025-09-07T08:12:53.4158229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4158296Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4158522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4158619Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4158833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4158932Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4159168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4159237Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4159469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4159537Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4159761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4159857Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4159862Z 
2025-09-07T08:12:53.4159952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4160168Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4160228Z     return mod(**inputs)
2025-09-07T08:12:53.4160461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4160556Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4160793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4160860Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4161074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4161148Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4161376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4161447Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4161679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4161750Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4161982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4162051Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4162054Z 
2025-09-07T08:12:53.4162126Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4162221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4162406Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4162467Z     return mod(**inputs)
2025-09-07T08:12:53.4162699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4162761Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4162998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4163059Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4163274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4163340Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4163573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4163644Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4163870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:12:53.4163973Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4164200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4164326Z     return self.weight * hidden_states
2025-09-07T08:12:53.4164331Z 
2025-09-07T08:12:53.4164419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4164602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4164668Z     return mod(**inputs)
2025-09-07T08:12:53.4164898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4164965Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4165195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4165258Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4165475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4165541Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4165807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4165879Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4166142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4166217Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4166444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4166518Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4166521Z 
2025-09-07T08:12:53.4166610Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4166794Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4166850Z     return mod(**inputs)
2025-09-07T08:12:53.4167082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4167157Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4167384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4167450Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4167660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4167726Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4167955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4168025Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4168252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4168324Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4168553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4168621Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4168624Z 
2025-09-07T08:12:53.4168713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4168900Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4168957Z     return mod(**inputs)
2025-09-07T08:12:53.4169189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4169252Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4169481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4169545Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4169757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4169889Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4170115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4170186Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4170414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4170486Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4170713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4170826Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4170830Z 
2025-09-07T08:12:53.4170920Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4171099Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4171157Z     return mod(**inputs)
2025-09-07T08:12:53.4171423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4171486Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4171756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4171817Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4172028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4172100Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4172329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4172402Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4172630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4172713Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4172938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4173050Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4173053Z 
2025-09-07T08:12:53.4173147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4173328Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4173386Z     return mod(**inputs)
2025-09-07T08:12:53.4173615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4173677Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4173909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4173974Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4174190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4174254Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4174481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4174556Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4174779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4174857Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4175081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4175193Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4175242Z 
2025-09-07T08:12:53.4175331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4175546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4175605Z     return mod(**inputs)
2025-09-07T08:12:53.4175838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4175905Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4176132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4176191Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4176408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4176473Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4176701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4176804Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4177040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4177111Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4177367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4177518Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4177522Z 
2025-09-07T08:12:53.4177645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4177832Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4177887Z     return mod(**inputs)
2025-09-07T08:12:53.4178116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4178190Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4178421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4178486Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4178701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4178767Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4178999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4179066Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4179296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4179367Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4179596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4179669Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4179672Z 
2025-09-07T08:12:53.4179759Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4179948Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4180003Z     return mod(**inputs)
2025-09-07T08:12:53.4180237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4180300Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4180529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4180592Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4180804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4180914Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4181180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4181253Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4181485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4181556Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4181788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4181883Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4181886Z 
2025-09-07T08:12:53.4181987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4182168Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4182226Z     return mod(**inputs)
2025-09-07T08:12:53.4182498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4182563Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4182822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4182885Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4183098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4183171Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4183397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4183471Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4183698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4183778Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4184007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4184098Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4184102Z 
2025-09-07T08:12:53.4184198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4184382Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4184443Z     return mod(**inputs)
2025-09-07T08:12:53.4184670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4184733Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4184969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4185034Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4185250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4185317Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4185547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4185622Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4185848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4185923Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4186149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4186245Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4186247Z 
2025-09-07T08:12:53.4186334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4186550Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4186642Z     return mod(**inputs)
2025-09-07T08:12:53.4186874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4186942Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4187169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4187231Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4187450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4187519Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4187747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4187818Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4188079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4188156Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4188416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4188516Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4188520Z 
2025-09-07T08:12:53.4188610Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4188798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4188855Z     return mod(**inputs)
2025-09-07T08:12:53.4189086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4189153Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4189384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4189451Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4189665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4189736Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4189963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4190034Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4190268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4190340Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4190566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4190640Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4190645Z 
2025-09-07T08:12:53.4190717Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4190808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4190992Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4191054Z     return mod(**inputs)
2025-09-07T08:12:53.4191283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4191344Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4191580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4191640Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4191858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4191962Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4192223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4192312Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4192540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4192633Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4192856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4192928Z     return self.weight * hidden_states
2025-09-07T08:12:53.4192932Z 
2025-09-07T08:12:53.4193023Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4193212Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4193276Z     return mod(**inputs)
2025-09-07T08:12:53.4193538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4193606Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4193838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4193948Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4194170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4194238Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4194471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4194547Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4194772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4194887Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4195117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4195191Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4195194Z 
2025-09-07T08:12:53.4195288Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4195478Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4195535Z     return mod(**inputs)
2025-09-07T08:12:53.4195792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4195874Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4196103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4196167Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4196382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4196449Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4196683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4196763Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4196992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4197093Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4197325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4197402Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4197405Z 
2025-09-07T08:12:53.4197493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4197719Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4197848Z     return mod(**inputs)
2025-09-07T08:12:53.4198083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4198148Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4198381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4198447Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4198660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4198730Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4198958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4199036Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4199299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4199399Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4199665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4199738Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4199741Z 
2025-09-07T08:12:53.4199815Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4199902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4200083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4200145Z     return mod(**inputs)
2025-09-07T08:12:53.4200377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4200445Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4200676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4200739Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4200960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4201026Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4201255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4201330Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4201559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4201660Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4201884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4201960Z     return self.weight * hidden_states
2025-09-07T08:12:53.4201964Z 
2025-09-07T08:12:53.4202052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4202239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4202296Z     return mod(**inputs)
2025-09-07T08:12:53.4202523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4202591Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4202818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4202879Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4203218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4203288Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4203572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4203689Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4203921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4203996Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4204223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4204295Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4204299Z 
2025-09-07T08:12:53.4204388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4204574Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4204634Z     return mod(**inputs)
2025-09-07T08:12:53.4204867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4204971Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4205207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4205313Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4205533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4205604Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4205840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4205914Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4206146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4206222Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4206459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4206526Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4206530Z 
2025-09-07T08:12:53.4206632Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4206825Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4206883Z     return mod(**inputs)
2025-09-07T08:12:53.4207127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4207193Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4207429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4207493Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4207710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4207789Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4208018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4208096Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4208326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4208400Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4208632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4208755Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4208758Z 
2025-09-07T08:12:53.4208858Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4209045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4209170Z     return mod(**inputs)
2025-09-07T08:12:53.4209407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4209471Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4209710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4209772Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4209996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4210064Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4210295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4210375Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4210604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4210717Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4210947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4211088Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4211098Z 
2025-09-07T08:12:53.4211190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4211376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4211440Z     return mod(**inputs)
2025-09-07T08:12:53.4211671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4211741Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4211979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4212047Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4212269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4212337Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4212570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4212639Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4212867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4212946Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4213174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4213290Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4213295Z 
2025-09-07T08:12:53.4213384Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4213577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4213635Z     return mod(**inputs)
2025-09-07T08:12:53.4213866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4213936Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4214166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4214234Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4214445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4214512Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4214745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4214875Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4215110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4215180Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4215407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4215562Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4215566Z 
2025-09-07T08:12:53.4215653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4215843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4215900Z     return mod(**inputs)
2025-09-07T08:12:53.4216135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4216201Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4216480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4216550Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4216794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4216869Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4217099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4217170Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4217406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4217476Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4217763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4217839Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4217842Z 
2025-09-07T08:12:53.4217938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4218122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4218179Z     return mod(**inputs)
2025-09-07T08:12:53.4218429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4218491Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4218728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4218793Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4219008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4219087Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4219320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4219396Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4219626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4219707Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4219945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4220046Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4220049Z 
2025-09-07T08:12:53.4220147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4220331Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4220429Z     return mod(**inputs)
2025-09-07T08:12:53.4220665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4220760Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4221001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4221063Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4221284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4221353Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4221583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4221660Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4221889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4221969Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4222232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4222330Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4222339Z 
2025-09-07T08:12:53.4222457Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4222643Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4222711Z     return mod(**inputs)
2025-09-07T08:12:53.4222945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4223018Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4223254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4223314Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4223539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4223607Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4223841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4223910Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4224137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4224211Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4224438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4224535Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4224538Z 
2025-09-07T08:12:53.4224624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4224812Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4224871Z     return mod(**inputs)
2025-09-07T08:12:53.4225101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4225170Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4225416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4225483Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4225696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4225763Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4225992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4226062Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4226331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4226431Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4226663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4226758Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4226762Z 
2025-09-07T08:12:53.4226848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4227036Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4227093Z     return mod(**inputs)
2025-09-07T08:12:53.4227327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4227387Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4227620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4227717Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4227933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4228035Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4228268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4228339Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4228570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4228640Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4228871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4228940Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4228943Z 
2025-09-07T08:12:53.4229033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4229220Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4229277Z     return mod(**inputs)
2025-09-07T08:12:53.4229508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4229572Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4229804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4229864Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4230073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4230141Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4230365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4230440Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4230666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:12:53.4230789Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:12:53.4230793Z 
2025-09-07T08:12:53.4230866Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4230954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4231140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4231195Z     return mod(**inputs)
2025-09-07T08:12:53.4231425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4231489Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4231752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4231849Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4232064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4232134Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4232359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4232430Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4232658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:12:53.4232755Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4232985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4233054Z     return self.weight * hidden_states
2025-09-07T08:12:53.4233058Z 
2025-09-07T08:12:53.4233178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4233364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4233419Z     return mod(**inputs)
2025-09-07T08:12:53.4233681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4233746Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4233974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4234038Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4234246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4234315Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4234539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4234617Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4234843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4234915Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4235145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4235212Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4235215Z 
2025-09-07T08:12:53.4235313Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4235494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4235548Z     return mod(**inputs)
2025-09-07T08:12:53.4235778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4235843Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4236074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4236133Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4236347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4236415Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4236643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4236720Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4236945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4237021Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4237247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4237392Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4237395Z 
2025-09-07T08:12:53.4237486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4237667Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4237725Z     return mod(**inputs)
2025-09-07T08:12:53.4237953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4238015Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4238247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4238308Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4238523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4238592Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4238854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4238927Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4239182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4239261Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4239490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4239612Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4239615Z 
2025-09-07T08:12:53.4239705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4239887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4239951Z     return mod(**inputs)
2025-09-07T08:12:53.4240185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4240253Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4240483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4240545Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4240757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4240826Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4241055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4241122Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4241351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4241425Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4241653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4241765Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4241770Z 
2025-09-07T08:12:53.4241859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4242045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4242102Z     return mod(**inputs)
2025-09-07T08:12:53.4242333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4242403Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4242636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4242735Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4242979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4243048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4243279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4243348Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4243575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4243646Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4243879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4243992Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4243996Z 
2025-09-07T08:12:53.4244088Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4244311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4244370Z     return mod(**inputs)
2025-09-07T08:12:53.4244638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4244703Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4244938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4245007Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4245219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4245290Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4245515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4245592Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4245823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4245897Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4246127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4246274Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4246278Z 
2025-09-07T08:12:53.4246371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4246553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4246611Z     return mod(**inputs)
2025-09-07T08:12:53.4246844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4246906Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4247140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4247199Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4247408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4247477Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4247702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4247783Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4248010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4248089Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4248314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4248448Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4248452Z 
2025-09-07T08:12:53.4248549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4248731Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4248799Z     return mod(**inputs)
2025-09-07T08:12:53.4249031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4249092Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4249328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4249391Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4249608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4249674Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4249935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4250013Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4250270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4250351Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4250578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4250679Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4250682Z 
2025-09-07T08:12:53.4250771Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4250954Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4251019Z     return mod(**inputs)
2025-09-07T08:12:53.4251254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4251324Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4251555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4251622Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4251842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4251909Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4252142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4252211Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4252439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4252513Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4252740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4252834Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4252837Z 
2025-09-07T08:12:53.4252924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4253112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4253168Z     return mod(**inputs)
2025-09-07T08:12:53.4253398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4253468Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4253700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4253768Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4254020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4254120Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4254364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4254440Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4254675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4254749Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4254987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4255081Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4255086Z 
2025-09-07T08:12:53.4255176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4255369Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4255458Z     return mod(**inputs)
2025-09-07T08:12:53.4255695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4255759Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4256021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4256092Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4256318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4256394Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4256643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4256714Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4256952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4257027Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4257260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4257354Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4257359Z 
2025-09-07T08:12:53.4257454Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4257695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4257754Z     return mod(**inputs)
2025-09-07T08:12:53.4257994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4258057Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4258290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4258357Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4258568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4258640Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4258871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4258948Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4259176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4259258Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4259486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4259554Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4259606Z 
2025-09-07T08:12:53.4259681Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4259800Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4259987Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4260041Z     return mod(**inputs)
2025-09-07T08:12:53.4260271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4260339Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4260567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4260636Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4260849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4260915Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4261185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4261271Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4261509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4261629Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4261866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4261936Z     return self.weight * hidden_states
2025-09-07T08:12:53.4261939Z 
2025-09-07T08:12:53.4262032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4262224Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4262282Z     return mod(**inputs)
2025-09-07T08:12:53.4262520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4262588Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4262817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4262882Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4263095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4263167Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4263392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4263468Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4263696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4263802Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4264039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4264111Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4264114Z 
2025-09-07T08:12:53.4264203Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4264385Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4264440Z     return mod(**inputs)
2025-09-07T08:12:53.4264670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4264731Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4264961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4265020Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4265236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4265372Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4265601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4265687Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4265915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4266018Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4266254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4266328Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4266331Z 
2025-09-07T08:12:53.4266429Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4266611Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4266676Z     return mod(**inputs)
2025-09-07T08:12:53.4266936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4266998Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4267261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4267321Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4267537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4267603Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4267832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4267910Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4268134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4268240Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4268464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4268540Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4268544Z 
2025-09-07T08:12:53.4268615Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4268706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4268898Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4268954Z     return mod(**inputs)
2025-09-07T08:12:53.4269191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4269254Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4269488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4269563Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4269776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4269852Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4270079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4270154Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4270387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4270483Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4270719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4270821Z     return self.weight * hidden_states
2025-09-07T08:12:53.4270851Z 
2025-09-07T08:12:53.4270944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4271129Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4271184Z     return mod(**inputs)
2025-09-07T08:12:53.4271418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4271478Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4271719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4271785Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4271994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4272065Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4272295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4272405Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4272642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4272752Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4273001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4273070Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4273074Z 
2025-09-07T08:12:53.4273174Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4273356Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4273429Z     return mod(**inputs)
2025-09-07T08:12:53.4273661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4273729Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4273969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4274031Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4274251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4274325Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4274561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4274638Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4274865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4274938Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4275160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4275234Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4275241Z 
2025-09-07T08:12:53.4275332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4275516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4275573Z     return mod(**inputs)
2025-09-07T08:12:53.4275800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4275864Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4276092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4276151Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4276361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4276459Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4276717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4276785Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4277012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4277085Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4277310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4277431Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4277435Z 
2025-09-07T08:12:53.4277521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4277705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4277765Z     return mod(**inputs)
2025-09-07T08:12:53.4278027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4278092Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4278369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4278432Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4278646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4278716Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4278954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4279025Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4279260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4279336Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4279567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4279685Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4279689Z 
2025-09-07T08:12:53.4279779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4279969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4280027Z     return mod(**inputs)
2025-09-07T08:12:53.4280265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4280326Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4280553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4280617Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4280830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4280896Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4281125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4281194Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4281421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4281491Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4281720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4281828Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4281832Z 
2025-09-07T08:12:53.4281957Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4282166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4282221Z     return mod(**inputs)
2025-09-07T08:12:53.4282453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4282515Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4282743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4282802Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4283013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4283080Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4283305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4283377Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4283645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4283721Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4283988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4284138Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4284142Z 
2025-09-07T08:12:53.4284241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4284434Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4284493Z     return mod(**inputs)
2025-09-07T08:12:53.4284727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4284793Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4285027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4285091Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4285315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4285386Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4285614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4285688Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4285914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4285985Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4286209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4286278Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4286286Z 
2025-09-07T08:12:53.4286377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4286563Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4286623Z     return mod(**inputs)
2025-09-07T08:12:53.4286852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4286916Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4287149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4287209Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4287423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4287491Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4287803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4287903Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4288130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4288205Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4288429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4288535Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4288539Z 
2025-09-07T08:12:53.4288629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4288819Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4288877Z     return mod(**inputs)
2025-09-07T08:12:53.4289109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4289208Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4289444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4289543Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4289759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4289828Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4290065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4290136Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4290370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4290442Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4290674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4290778Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4290782Z 
2025-09-07T08:12:53.4290873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4291064Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4291121Z     return mod(**inputs)
2025-09-07T08:12:53.4291360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4291426Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4291657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4291726Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4291940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4292019Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4292247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4292318Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4292550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4292622Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4292857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4292946Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4292951Z 
2025-09-07T08:12:53.4293040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4293221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4293344Z     return mod(**inputs)
2025-09-07T08:12:53.4293576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4293637Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4293868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4293928Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4294138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4294205Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4294430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4294500Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4294726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4294826Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4295057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4295179Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4295182Z 
2025-09-07T08:12:53.4295272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4295452Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4295510Z     return mod(**inputs)
2025-09-07T08:12:53.4295739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4295799Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4296029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4296092Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4296305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4296369Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4296595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4296665Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4296890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4296959Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4297182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4297247Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4297254Z 
2025-09-07T08:12:53.4297325Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4297414Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4297667Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4297722Z     return mod(**inputs)
2025-09-07T08:12:53.4297957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4298018Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4298246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4298308Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4298518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4298584Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4298808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4298942Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4299171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:12:53.4299269Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4299506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4299575Z     return self.weight * hidden_states
2025-09-07T08:12:53.4299578Z 
2025-09-07T08:12:53.4299666Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4299858Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4299914Z     return mod(**inputs)
2025-09-07T08:12:53.4300154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4307286Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4307707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4307782Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4308088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4308165Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4308418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4308493Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4308744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4308827Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4309074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4309154Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4309159Z 
2025-09-07T08:12:53.4309266Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4309475Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4309536Z     return mod(**inputs)
2025-09-07T08:12:53.4309796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4309869Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4310099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4310169Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4310390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4310471Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4310711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4310786Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4311018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4311091Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4311319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4311389Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4311393Z 
2025-09-07T08:12:53.4311489Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4311690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4311798Z     return mod(**inputs)
2025-09-07T08:12:53.4312041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4312155Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4312390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4312461Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4312678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4312753Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4312983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4313056Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4313287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4313364Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4313631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4313759Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4313790Z 
2025-09-07T08:12:53.4313893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4314091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4314154Z     return mod(**inputs)
2025-09-07T08:12:53.4314404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4314469Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4314706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4314771Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4314994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4315073Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4315311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4315386Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4315612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4315687Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4315916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4316031Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4316035Z 
2025-09-07T08:12:53.4316129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4316317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4316381Z     return mod(**inputs)
2025-09-07T08:12:53.4316613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4316685Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4316917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4316979Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4317198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4317265Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4317491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4317603Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4317862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4317938Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4318169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4318282Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4318286Z 
2025-09-07T08:12:53.4318376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4318566Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4318624Z     return mod(**inputs)
2025-09-07T08:12:53.4318852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4318915Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4319176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4319238Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4319453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4319551Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4319784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4319853Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4320079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4320151Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4320382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4320545Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4320551Z 
2025-09-07T08:12:53.4320642Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4320836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4320893Z     return mod(**inputs)
2025-09-07T08:12:53.4321129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4321194Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4321425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4321487Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4321704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4321773Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4322007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4322077Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4322311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4322382Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4322611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4322680Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4322684Z 
2025-09-07T08:12:53.4322772Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4322959Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4323014Z     return mod(**inputs)
2025-09-07T08:12:53.4323284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4323395Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4323627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4323691Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4323903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4323973Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4324196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4324265Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4324491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4324565Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4324830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4324936Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4324940Z 
2025-09-07T08:12:53.4325072Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4325266Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4325334Z     return mod(**inputs)
2025-09-07T08:12:53.4325582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4325652Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4325885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4325947Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4326164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4326245Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4326479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4326556Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4326784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4326856Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4327087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4327180Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4327184Z 
2025-09-07T08:12:53.4327278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4327460Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4327523Z     return mod(**inputs)
2025-09-07T08:12:53.4327754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4327818Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4328054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4328116Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4328335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4328403Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4328629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4328703Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4328968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4329097Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4329324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4329426Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4329429Z 
2025-09-07T08:12:53.4329525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4329710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4329772Z     return mod(**inputs)
2025-09-07T08:12:53.4330002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4330072Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4330302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4330400Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4330622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4330691Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4330958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4331029Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4331258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4331330Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4331556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4331651Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4331658Z 
2025-09-07T08:12:53.4331748Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4331931Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4331986Z     return mod(**inputs)
2025-09-07T08:12:53.4332216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4332279Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4332507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4332571Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4332781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4332845Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4333073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4333146Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4333372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4333443Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4333666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4333737Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4333740Z 
2025-09-07T08:12:53.4333827Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4334009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4334063Z     return mod(**inputs)
2025-09-07T08:12:53.4334292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4334391Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4334651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4334717Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4334931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4335000Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4335228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4335301Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4335533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T08:12:53.4335656Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:12:53.4335660Z 
2025-09-07T08:12:53.4335743Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4335871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4336065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4336126Z     return mod(**inputs)
2025-09-07T08:12:53.4336389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4336463Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4336695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4336765Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4336980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4337049Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4337284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4337371Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4337694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4337784Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4338015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4338092Z     return self.weight * hidden_states
2025-09-07T08:12:53.4338096Z 
2025-09-07T08:12:53.4338187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4338380Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4338438Z     return mod(**inputs)
2025-09-07T08:12:53.4338678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4338744Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4338978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4339049Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4339265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4339338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4339566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4339646Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4339880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4339994Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4340268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4340377Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4340380Z 
2025-09-07T08:12:53.4340471Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4340661Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4340719Z     return mod(**inputs)
2025-09-07T08:12:53.4340957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4341021Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4341257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4341319Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4341532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4341609Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4341868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4341949Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4342207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4342311Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4342545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4342617Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4342621Z 
2025-09-07T08:12:53.4342716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4342902Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4342968Z     return mod(**inputs)
2025-09-07T08:12:53.4343202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4343267Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4343507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4343570Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4343786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4343853Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4344081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4344159Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4344387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4344493Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4344717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4344785Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4344793Z 
2025-09-07T08:12:53.4344865Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4344952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4345136Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4345191Z     return mod(**inputs)
2025-09-07T08:12:53.4345426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4345487Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4345715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4345846Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4346060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4346132Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4346364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4346435Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4346663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4346761Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4346988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4347055Z     return self.weight * hidden_states
2025-09-07T08:12:53.4347061Z 
2025-09-07T08:12:53.4347148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4347368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4347425Z     return mod(**inputs)
2025-09-07T08:12:53.4347688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4347750Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4347983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4348046Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4348257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4348326Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4348552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4348629Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4348854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4348925Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4349153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4349228Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4349231Z 
2025-09-07T08:12:53.4349322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4349511Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4349569Z     return mod(**inputs)
2025-09-07T08:12:53.4349798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4349863Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4350093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4350159Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4350373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4350442Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4350674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4350745Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4350977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4351048Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4351276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4351435Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4351440Z 
2025-09-07T08:12:53.4351529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4351716Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4351772Z     return mod(**inputs)
2025-09-07T08:12:53.4352006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4352067Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4352297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4352365Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4352577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4352649Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4352909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4352984Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4353248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4353321Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4353551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4353672Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4353676Z 
2025-09-07T08:12:53.4353769Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4353953Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4354010Z     return mod(**inputs)
2025-09-07T08:12:53.4354247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4354316Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4354549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4354611Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4354825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4354895Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4355123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4355194Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4355419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4355490Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4355721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4355834Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4355837Z 
2025-09-07T08:12:53.4355933Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4356116Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4356175Z     return mod(**inputs)
2025-09-07T08:12:53.4356411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4356478Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4356708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4356767Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4357018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4357111Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4357337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4357408Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4357632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4357703Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4357930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4358036Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4358042Z 
2025-09-07T08:12:53.4358128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4358309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4358399Z     return mod(**inputs)
2025-09-07T08:12:53.4358634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4358730Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4358964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4359025Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4359239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4359306Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4359552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4359625Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4359853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4359930Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4360156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4360305Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4360309Z 
2025-09-07T08:12:53.4360398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4360582Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4360637Z     return mod(**inputs)
2025-09-07T08:12:53.4360867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4360933Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4361167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4361233Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4361442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4361508Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4361735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4361803Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4362030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4362098Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4362322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4362426Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4362458Z 
2025-09-07T08:12:53.4362547Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4362730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4362784Z     return mod(**inputs)
2025-09-07T08:12:53.4363017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4363079Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4363308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4363370Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4363579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4363646Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4363870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4363971Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4364202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4364298Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4364532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4364634Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4364637Z 
2025-09-07T08:12:53.4364726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4364915Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4364971Z     return mod(**inputs)
2025-09-07T08:12:53.4365208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4365272Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4365503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4365563Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4365775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4365842Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4366070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4366143Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4366373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4366443Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4366676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4366772Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4366775Z 
2025-09-07T08:12:53.4366863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4367046Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4367102Z     return mod(**inputs)
2025-09-07T08:12:53.4367331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4367392Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4367622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4367680Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4367891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4368014Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4368254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4368324Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4368552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4368623Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4368847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4368938Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4368943Z 
2025-09-07T08:12:53.4369033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4369219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4369280Z     return mod(**inputs)
2025-09-07T08:12:53.4369545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4369614Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4369889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4369950Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4370163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4370228Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4370457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4370525Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4370750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4370826Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4371050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4371146Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4371151Z 
2025-09-07T08:12:53.4371239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4371424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4371481Z     return mod(**inputs)
2025-09-07T08:12:53.4371708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4371770Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4371998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4372060Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4372273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4372338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4372570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4372644Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4372880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4372948Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4373171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4373240Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4373244Z 
2025-09-07T08:12:53.4373314Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4373644Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4373859Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4373915Z     return mod(**inputs)
2025-09-07T08:12:53.4374148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4374209Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4374444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4374504Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4374716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4374780Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4375005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4375082Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4375341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:12:53.4375442Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4375699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4375768Z     return self.weight * hidden_states
2025-09-07T08:12:53.4375771Z 
2025-09-07T08:12:53.4375861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4376041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4376097Z     return mod(**inputs)
2025-09-07T08:12:53.4376327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4376392Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4376625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4376687Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4376900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4376964Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4377193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4377263Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4377489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4377633Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4377860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4377938Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4377942Z 
2025-09-07T08:12:53.4378028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4378211Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4378277Z     return mod(**inputs)
2025-09-07T08:12:53.4378504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4378567Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4378793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4378855Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4379084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4379195Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4379459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4379529Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4379760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4379830Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4380052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4380120Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4380124Z 
2025-09-07T08:12:53.4380213Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4380395Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4380453Z     return mod(**inputs)
2025-09-07T08:12:53.4380683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4380779Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4381012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4381158Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4381372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4381439Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4381666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4381733Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4381960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4382033Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4382264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4382381Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4382384Z 
2025-09-07T08:12:53.4382473Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4382656Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4382711Z     return mod(**inputs)
2025-09-07T08:12:53.4382941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4383002Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4383231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4383292Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4383503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4383574Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4383799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4383874Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4384099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4384172Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4384399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4384509Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4384512Z 
2025-09-07T08:12:53.4384603Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4384819Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4384907Z     return mod(**inputs)
2025-09-07T08:12:53.4385147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4385210Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4385442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4385506Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4385718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4385791Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4386015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4386091Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4386350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4386429Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4386687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4386799Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4386802Z 
2025-09-07T08:12:53.4386894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4387076Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4387132Z     return mod(**inputs)
2025-09-07T08:12:53.4387365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4387429Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4387659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4387725Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4387937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4388004Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4388230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4388297Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4388522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4388593Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4388817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4388961Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4388967Z 
2025-09-07T08:12:53.4389056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4389237Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4389298Z     return mod(**inputs)
2025-09-07T08:12:53.4389530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4389596Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4389826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4389889Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4390103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4390168Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4390431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4390532Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4390762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4390836Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4391066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4391137Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4391140Z 
2025-09-07T08:12:53.4391227Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4391416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4391472Z     return mod(**inputs)
2025-09-07T08:12:53.4391701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4391809Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4392040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4392103Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4392343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4392411Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4392638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4392706Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4392933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4393003Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4393233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4393328Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4393332Z 
2025-09-07T08:12:53.4393419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4393602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4393656Z     return mod(**inputs)
2025-09-07T08:12:53.4393887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4393947Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4394175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4394237Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4394446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4394517Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4394748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4394823Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4395051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4395123Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4395355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4395445Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4395448Z 
2025-09-07T08:12:53.4395536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4395716Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4395804Z     return mod(**inputs)
2025-09-07T08:12:53.4396069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4396131Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4396364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4396424Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4396635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4396702Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4396926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4396997Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4397222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4397328Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4397555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4397674Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4397678Z 
2025-09-07T08:12:53.4397767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4397948Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4398005Z     return mod(**inputs)
2025-09-07T08:12:53.4398233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4398294Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4398524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4398588Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4398806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4398873Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4399108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4399177Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4399401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4399475Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4399700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4399796Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4399799Z 
2025-09-07T08:12:53.4399889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4400075Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4400133Z     return mod(**inputs)
2025-09-07T08:12:53.4400365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4400431Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4400657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4400716Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4400930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4400997Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4401228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4401336Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4401596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4401670Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4401896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4401969Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4401972Z 
2025-09-07T08:12:53.4402042Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4402131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4402311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4402365Z     return mod(**inputs)
2025-09-07T08:12:53.4402595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4402657Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4402921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4402982Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4403352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4403421Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4403656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4403742Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4403966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4404052Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4404277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4404349Z     return self.weight * hidden_states
2025-09-07T08:12:53.4404353Z 
2025-09-07T08:12:53.4404445Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4404625Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4404681Z     return mod(**inputs)
2025-09-07T08:12:53.4404912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4404974Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4405202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4405262Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4405474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4405540Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4405769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4405846Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4406073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4406184Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4406415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4406485Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4406488Z 
2025-09-07T08:12:53.4406575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4406757Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4406860Z     return mod(**inputs)
2025-09-07T08:12:53.4407094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4407202Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4407435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4407495Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4407707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4407772Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4407999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4408073Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4408300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4408405Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4408674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4408748Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4408783Z 
2025-09-07T08:12:53.4408871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4409054Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4409108Z     return mod(**inputs)
2025-09-07T08:12:53.4409335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4409397Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4409624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4409687Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4409899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4409965Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4410208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4410282Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4410509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4410606Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4410832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4410899Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4410903Z 
2025-09-07T08:12:53.4410989Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4411177Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4411232Z     return mod(**inputs)
2025-09-07T08:12:53.4411461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4411523Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4411753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4411815Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4412024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4412091Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4412320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4412427Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4412703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T08:12:53.4412822Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T08:12:53.4412825Z 
2025-09-07T08:12:53.4412897Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4412985Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4413168Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4413222Z     return mod(**inputs)
2025-09-07T08:12:53.4413452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4413515Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4413742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4413805Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4414047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4414112Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4414372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4414445Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4414674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:12:53.4414768Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4414994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4415059Z     return self.weight * hidden_states
2025-09-07T08:12:53.4415063Z 
2025-09-07T08:12:53.4415152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4415337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4415391Z     return mod(**inputs)
2025-09-07T08:12:53.4415622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4415683Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4415909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4415970Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4416179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4416245Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4416470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4416541Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4416772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4416843Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4417070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4417137Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4417140Z 
2025-09-07T08:12:53.4417228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4417409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4417463Z     return mod(**inputs)
2025-09-07T08:12:53.4417751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4417812Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4418086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4418176Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4418386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4418455Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4418681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4418752Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4418977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4419046Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4419274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4419342Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4419347Z 
2025-09-07T08:12:53.4419471Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4419653Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4419709Z     return mod(**inputs)
2025-09-07T08:12:53.4419968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4420031Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4420259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4420319Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4420530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4420594Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4420822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4420895Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4421119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4421191Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4421414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4421528Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4421534Z 
2025-09-07T08:12:53.4421621Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4421801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4421858Z     return mod(**inputs)
2025-09-07T08:12:53.4422088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4422155Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4422381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4422441Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4422653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4422717Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4422944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4423010Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4423234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4423304Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4423568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4423712Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4423715Z 
2025-09-07T08:12:53.4423804Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4423996Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4424050Z     return mod(**inputs)
2025-09-07T08:12:53.4424277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4424342Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4424569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4424631Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4424840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4424938Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4425169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4425267Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4425498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4425567Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4425792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4425900Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4425904Z 
2025-09-07T08:12:53.4425990Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4426175Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4426234Z     return mod(**inputs)
2025-09-07T08:12:53.4426466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4426530Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4426760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4426824Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4427033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4427102Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4427328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4427397Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4427624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4427700Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4427931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4428077Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4428081Z 
2025-09-07T08:12:53.4428175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4428356Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4428411Z     return mod(**inputs)
2025-09-07T08:12:53.4428651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4428714Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4428953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4429074Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4429288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4429357Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4429587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4429665Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4429890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4429958Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4430186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4430256Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4430261Z 
2025-09-07T08:12:53.4430356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4430566Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4430625Z     return mod(**inputs)
2025-09-07T08:12:53.4430884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4430947Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4431178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4431237Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4431449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4431517Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4431742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4431816Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4432041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4432115Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4432338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4432438Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4432446Z 
2025-09-07T08:12:53.4432536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4432718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4432777Z     return mod(**inputs)
2025-09-07T08:12:53.4433004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4433072Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4433301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4433362Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4433580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4433646Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4433875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4433946Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4434172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4434247Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4434472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4434645Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4434649Z 
2025-09-07T08:12:53.4434737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4434926Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4434981Z     return mod(**inputs)
2025-09-07T08:12:53.4435209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4435274Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4435504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4435567Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4435779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4435847Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4436109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4436180Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4436444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4436513Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4436743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4436834Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4436838Z 
2025-09-07T08:12:53.4436926Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4437111Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4437166Z     return mod(**inputs)
2025-09-07T08:12:53.4437403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4437466Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4437695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4437756Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4437967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4438036Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4438267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4438335Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4438564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4438635Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4438864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4438957Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4438960Z 
2025-09-07T08:12:53.4439052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4439236Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4439290Z     return mod(**inputs)
2025-09-07T08:12:53.4439522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4439584Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4439812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4439873Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4440124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4440223Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4440451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:12:53.4440522Z     self_attention_outputs = self.layer[0](
2025-09-07T08:12:53.4440748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:12:53.4440817Z     attention_output = self.SelfAttention(
2025-09-07T08:12:53.4441047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4441113Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4441117Z 
2025-09-07T08:12:53.4441191Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4441280Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4441495Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4441557Z     return mod(**inputs)
2025-09-07T08:12:53.4441785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4441983Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4442217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4442287Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4442500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4442566Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4442797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4442867Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4443107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:12:53.4443209Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4443437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4443511Z     return self.weight * hidden_states
2025-09-07T08:12:53.4443514Z 
2025-09-07T08:12:53.4443606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4443798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4443854Z     return mod(**inputs)
2025-09-07T08:12:53.4444085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4444155Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4444388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4444458Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4444671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4444747Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4444978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4445048Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4445280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4445353Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4445584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:12:53.4445686Z     query_states = self.q(hidden_states)
2025-09-07T08:12:53.4445717Z 
2025-09-07T08:12:53.4445811Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4446000Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4446057Z     return mod(**inputs)
2025-09-07T08:12:53.4446295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4446362Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4446599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4446663Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4446880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4446953Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4447213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4447292Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4447522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4447625Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4447859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:12:53.4447928Z     key_states = self.k(current_states)
2025-09-07T08:12:53.4447931Z 
2025-09-07T08:12:53.4448028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4448212Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4448269Z     return mod(**inputs)
2025-09-07T08:12:53.4448505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4448572Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4448805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4448864Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4449079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4449144Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4449368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4449437Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4449660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4449732Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4449958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4450072Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4450075Z 
2025-09-07T08:12:53.4450164Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4450346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4450403Z     return mod(**inputs)
2025-09-07T08:12:53.4450630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4450692Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4450921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4450980Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4451191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4451319Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4451548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4451617Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4451843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4451916Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4452140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4452250Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4452253Z 
2025-09-07T08:12:53.4452339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4452519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4452579Z     return mod(**inputs)
2025-09-07T08:12:53.4452840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4452904Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4453176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4453239Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4453449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4453513Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4453740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4453807Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4454032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4454107Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4454329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:12:53.4454450Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:12:53.4454453Z 
2025-09-07T08:12:53.4454540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4454721Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4454776Z     return mod(**inputs)
2025-09-07T08:12:53.4455003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4455068Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4455299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4455369Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4455585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4455657Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4455888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4455959Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4456193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4456264Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4456498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:12:53.4456640Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:12:53.4456678Z 
2025-09-07T08:12:53.4456795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4456985Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4457042Z     return mod(**inputs)
2025-09-07T08:12:53.4457281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4457346Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4457638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4457701Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4457914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4457985Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4458208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4458318Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4458549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4458622Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4458886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:12:53.4458956Z     value_states = self.v(current_states)
2025-09-07T08:12:53.4458959Z 
2025-09-07T08:12:53.4459051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4459234Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4459291Z     return mod(**inputs)
2025-09-07T08:12:53.4459523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4459587Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4459826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4459888Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4460108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4460176Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4460403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4460478Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4460705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4460782Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4461008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4461106Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4461110Z 
2025-09-07T08:12:53.4461205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4461386Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4461452Z     return mod(**inputs)
2025-09-07T08:12:53.4461681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4461744Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4461978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4462040Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4462258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4462358Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4462624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4462703Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4462930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4463005Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4463232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4463331Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4463334Z 
2025-09-07T08:12:53.4463424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4463606Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4463668Z     return mod(**inputs)
2025-09-07T08:12:53.4463927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4463999Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4464260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4464322Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4464558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4464627Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4464856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4464926Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4465160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4465235Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4465464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:12:53.4465560Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:12:53.4465563Z 
2025-09-07T08:12:53.4465653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4465841Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4465897Z     return mod(**inputs)
2025-09-07T08:12:53.4466128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4466195Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4466425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4466497Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4466709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4466779Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4467006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4467074Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4467304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4467373Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4467602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:12:53.4467697Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:12:53.4467700Z 
2025-09-07T08:12:53.4467789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4468010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4468099Z     return mod(**inputs)
2025-09-07T08:12:53.4468332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4468395Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4468626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4468692Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4468907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4468978Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4469202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:12:53.4469280Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:12:53.4469587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:12:53.4469661Z     attention_output = self.EncDecAttention(
2025-09-07T08:12:53.4469925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:12:53.4469995Z     attn_output = self.o(attn_output)
2025-09-07T08:12:53.4469998Z 
2025-09-07T08:12:53.4470074Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4470162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4470343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4470403Z     return mod(**inputs)
2025-09-07T08:12:53.4470630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4470699Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4470927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4470990Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4471207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4471273Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4471500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4471584Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4471816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:12:53.4471901Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:12:53.4472130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:12:53.4472206Z     return self.weight * hidden_states
2025-09-07T08:12:53.4472211Z 
2025-09-07T08:12:53.4472302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4472485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4472542Z     return mod(**inputs)
2025-09-07T08:12:53.4472788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4472859Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4473093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4473159Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4473371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4473438Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4473706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4473815Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4474047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4474154Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4474385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:12:53.4474456Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:12:53.4474460Z 
2025-09-07T08:12:53.4474550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4474735Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4474790Z     return mod(**inputs)
2025-09-07T08:12:53.4475069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4475135Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4475368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4475460Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4475676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4475751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4475977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4476058Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4476286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4476387Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4476621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:12:53.4476695Z     hidden_states = self.act(hidden_states)
2025-09-07T08:12:53.4476698Z 
2025-09-07T08:12:53.4476790Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4476970Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4477027Z     return mod(**inputs)
2025-09-07T08:12:53.4477262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:12:53.4477324Z     decoder_outputs = self.decoder(
2025-09-07T08:12:53.4477555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:12:53.4477617Z     layer_outputs = layer_module(
2025-09-07T08:12:53.4477831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:12:53.4477905Z     return super().__call__(*args, **kwargs)
2025-09-07T08:12:53.4478133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:12:53.4478214Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:12:53.4478439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:12:53.4478545Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:12:53.4478771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:12:53.4478840Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:12:53.4478844Z 
2025-09-07T08:12:53.4478919Z cudagraph partition due to non gpu ops
2025-09-07T08:12:53.4479038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4479256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4479313Z     return mod(**inputs)
2025-09-07T08:12:53.4479545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward
2025-09-07T08:12:53.4479658Z     sequence_output = sequence_output * (self.model_dim**-0.5)
2025-09-07T08:12:53.4479661Z 
2025-09-07T08:12:53.4479749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:12:53.4479939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:12:53.4479995Z     return mod(**inputs)
2025-09-07T08:12:53.4480224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward
2025-09-07T08:12:53.4480305Z     lm_logits = self.lm_head(sequence_output)
2025-09-07T08:12:53.4480309Z 
2025-09-07T08:13:14.2770757Z pass
2025-09-07T08:13:14.2771533Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:13:17.1665731Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:13:17.1669222Z   import pynvml  # type: ignore[import]
2025-09-07T08:13:19.1728476Z 
2025-09-07T08:13:28.6664800Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:13:28.6665064Z loading model: 0it [00:09, ?it/s]
2025-09-07T08:13:28.7036056Z cpu  eval  hf_T5_base                         
2025-09-07T08:13:43.0941899Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:13:49.6123556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:13:56.1018394Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:14:18.8250411Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8250744Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8251164Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8251506Z     return mod(**inputs)
2025-09-07T08:14:18.8251899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8252305Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8252669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8253042Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8253394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8253768Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8254156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8254532Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8254911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8255293Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8255663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
2025-09-07T08:14:18.8256075Z     position_bias = position_bias + causal_mask
2025-09-07T08:14:18.8256217Z 
2025-09-07T08:14:18.8256334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8256730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8257525Z     return mod(**inputs)
2025-09-07T08:14:18.8257878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8258384Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8258763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8259140Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8259487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8259856Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8260248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8260619Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8260990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8261400Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8261911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8262292Z     return self.weight * hidden_states
2025-09-07T08:14:18.8262423Z 
2025-09-07T08:14:18.8262602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8262969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8263311Z     return mod(**inputs)
2025-09-07T08:14:18.8263697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8264100Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8264472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8264849Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8265206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8265574Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8265950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8266334Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8266708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8267093Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8267471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8267839Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8267979Z 
2025-09-07T08:14:18.8268084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8276786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8277186Z     return mod(**inputs)
2025-09-07T08:14:18.8277587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8278002Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8278403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8278836Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8279362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8279740Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8280131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8280521Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8280974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8281400Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8281780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8282159Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8282290Z 
2025-09-07T08:14:18.8282403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8282769Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8283102Z     return mod(**inputs)
2025-09-07T08:14:18.8283452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8283837Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8284204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8284614Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8284964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8285332Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8285741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8286129Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8286488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8286870Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8287231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8287657Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8287845Z 
2025-09-07T08:14:18.8287957Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8288332Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8288656Z     return mod(**inputs)
2025-09-07T08:14:18.8289021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8289396Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8289748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8290115Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8290459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8290821Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8291185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8291563Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8291928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8292310Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8292673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8293077Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8293266Z 
2025-09-07T08:14:18.8293362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8293718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8294032Z     return mod(**inputs)
2025-09-07T08:14:18.8294364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8294812Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8295183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8295552Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8295888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8296241Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8296600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8296970Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8297394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8297776Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8298132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8298593Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8298780Z 
2025-09-07T08:14:18.8298877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8299273Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8299595Z     return mod(**inputs)
2025-09-07T08:14:18.8299926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8300298Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8300659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8301026Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8301367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8301723Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8302094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8302464Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8302835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8303377Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8303745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8304197Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8304412Z 
2025-09-07T08:14:18.8304510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8304865Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8305182Z     return mod(**inputs)
2025-09-07T08:14:18.8305529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8305901Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8306261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8306626Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8306954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8307311Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8307676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8308044Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8308397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8308912Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8309294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8309669Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8309798Z 
2025-09-07T08:14:18.8309905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8310257Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8310578Z     return mod(**inputs)
2025-09-07T08:14:18.8310934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8311310Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8311672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8312033Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8312427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8312798Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8313220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8313593Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8313957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8314338Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8314703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8315103Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8315271Z 
2025-09-07T08:14:18.8315371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8315719Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8316038Z     return mod(**inputs)
2025-09-07T08:14:18.8316381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8316755Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8317101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8317467Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8317807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8318163Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8318521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8318893Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8319260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8319636Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8320000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8320390Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8320552Z 
2025-09-07T08:14:18.8320644Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8320995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8321309Z     return mod(**inputs)
2025-09-07T08:14:18.8321647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8322005Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8322405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8322825Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8323155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8323516Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8323875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8324251Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8324611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8324983Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8325334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8325726Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8325952Z 
2025-09-07T08:14:18.8326046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8326403Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8326750Z     return mod(**inputs)
2025-09-07T08:14:18.8327092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8327476Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8327830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8328195Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8328518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8328876Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8329236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8329613Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8329968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8330331Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8330688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8331083Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8331235Z 
2025-09-07T08:14:18.8331336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8331680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8331990Z     return mod(**inputs)
2025-09-07T08:14:18.8332329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8332698Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8333048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8333403Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8333748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8334100Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8334462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8334823Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8335181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8335552Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8335961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8336367Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8336492Z 
2025-09-07T08:14:18.8336586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8336934Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8337306Z     return mod(**inputs)
2025-09-07T08:14:18.8337644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.8338011Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.8338360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8338725Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8339059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8339458Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8339821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.8340226Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.8340608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.8340985Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.8341346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8341711Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8341845Z 
2025-09-07T08:14:18.8341935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8342273Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8342591Z     return mod(**inputs)
2025-09-07T08:14:18.8342924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8343305Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8343659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8344027Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8344356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8344695Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8345062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8345438Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8345805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8346186Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8346539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8346910Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8347039Z 
2025-09-07T08:14:18.8347138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8347479Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8347792Z     return mod(**inputs)
2025-09-07T08:14:18.8348124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8348490Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8348848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8350502Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8350876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8351239Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8351635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8352010Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8352370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8352741Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8353125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8353494Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8353634Z 
2025-09-07T08:14:18.8353744Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8354127Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8354446Z     return mod(**inputs)
2025-09-07T08:14:18.8354791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8355191Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8355561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8355927Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8356261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8356633Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8356997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8357377Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8357734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8358106Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8358475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8358895Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8359074Z 
2025-09-07T08:14:18.8359177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8359519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8359831Z     return mod(**inputs)
2025-09-07T08:14:18.8360163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8360533Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8360888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8361249Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8361588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8361947Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8362308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8362674Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8363033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8363409Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8363768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8364225Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8364443Z 
2025-09-07T08:14:18.8364536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8364882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8365202Z     return mod(**inputs)
2025-09-07T08:14:18.8365530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8365899Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8366252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8366609Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8366945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8367294Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8367683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8368059Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8368452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8368831Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8369186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8369593Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8369776Z 
2025-09-07T08:14:18.8369869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8370220Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8370528Z     return mod(**inputs)
2025-09-07T08:14:18.8370853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8371227Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8371580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8371946Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8372278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8372624Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8372985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8373352Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8373713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8374076Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8374457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8374909Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8375119Z 
2025-09-07T08:14:18.8375221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8375558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8375885Z     return mod(**inputs)
2025-09-07T08:14:18.8376214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8376574Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8376925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8377326Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8377708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8378114Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8378481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8378850Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8379207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8379574Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8379934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8380376Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8380576Z 
2025-09-07T08:14:18.8380676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8381018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8381372Z     return mod(**inputs)
2025-09-07T08:14:18.8381704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8382100Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8382461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8382823Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8383160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8383508Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8383870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8384245Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8384598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8384978Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8385379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8385753Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8385889Z 
2025-09-07T08:14:18.8385984Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8386326Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8386644Z     return mod(**inputs)
2025-09-07T08:14:18.8386974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8387355Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8387696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8388066Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8388397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8388751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8389105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8389484Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8389842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8390235Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8390596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8390998Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8391208Z 
2025-09-07T08:14:18.8391332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8391676Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8391995Z     return mod(**inputs)
2025-09-07T08:14:18.8392325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8392693Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8393042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8393408Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8393744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8394091Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8394454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8394862Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8395225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8395635Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8395989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8396387Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8396547Z 
2025-09-07T08:14:18.8396643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8396981Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8397297Z     return mod(**inputs)
2025-09-07T08:14:18.8397619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8397984Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8398341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8398705Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8399034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8399382Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8399744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8400107Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8400471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8400830Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8401211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8401603Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8401756Z 
2025-09-07T08:14:18.8401864Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8402209Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8402531Z     return mod(**inputs)
2025-09-07T08:14:18.8402875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8403478Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8403843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8404219Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8404559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8404994Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8405415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8405798Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8406167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8406542Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8406916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8407318Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8407480Z 
2025-09-07T08:14:18.8407587Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8407936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8408264Z     return mod(**inputs)
2025-09-07T08:14:18.8408653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8409030Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8409443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8409822Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8410165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8410534Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8410906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8411281Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8411655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8412050Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8412440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8412808Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8412935Z 
2025-09-07T08:14:18.8413019Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8413252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8413604Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8413923Z     return mod(**inputs)
2025-09-07T08:14:18.8414263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8414639Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8414994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8415364Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8415703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8416057Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8416417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8416804Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8417289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8417679Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8418069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8418438Z     return self.weight * hidden_states
2025-09-07T08:14:18.8418614Z 
2025-09-07T08:14:18.8418722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8419110Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8419434Z     return mod(**inputs)
2025-09-07T08:14:18.8419783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8420153Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8420537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8420902Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8421239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8421601Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8421959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8422357Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8422763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8423181Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8423625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8423998Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8424125Z 
2025-09-07T08:14:18.8424226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8424570Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8424890Z     return mod(**inputs)
2025-09-07T08:14:18.8425234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8425598Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8425954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8426316Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8426654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8427013Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8427375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8427756Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8428128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8428546Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8428942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8429323Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8429450Z 
2025-09-07T08:14:18.8429545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8429895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8430207Z     return mod(**inputs)
2025-09-07T08:14:18.8430546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8430908Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8431253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8431617Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8431952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8432369Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8432763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8433139Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8433516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8433918Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8434309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8434676Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8434809Z 
2025-09-07T08:14:18.8434884Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8435107Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8435449Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8435762Z     return mod(**inputs)
2025-09-07T08:14:18.8436127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8436498Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8436889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8437257Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8437587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8437942Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8438307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8438683Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8439044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8439437Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8439825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8440192Z     return self.weight * hidden_states
2025-09-07T08:14:18.8440321Z 
2025-09-07T08:14:18.8440424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8440768Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8441075Z     return mod(**inputs)
2025-09-07T08:14:18.8441403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8441771Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8442125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8442486Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8442821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8443172Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8443532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8443901Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8444254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8444622Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8444987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8445350Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8445477Z 
2025-09-07T08:14:18.8445571Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8445995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8446312Z     return mod(**inputs)
2025-09-07T08:14:18.8446641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8447005Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8447352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8447716Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8448045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8448413Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8448787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8449160Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8449563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8449953Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8450355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8450720Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8450859Z 
2025-09-07T08:14:18.8450960Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8451324Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8451646Z     return mod(**inputs)
2025-09-07T08:14:18.8451987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8452353Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8452719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8453087Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8453430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8453779Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8454141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8454507Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8454870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8455241Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8455595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8456011Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8456201Z 
2025-09-07T08:14:18.8456300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8456649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8456964Z     return mod(**inputs)
2025-09-07T08:14:18.8457352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8457722Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8458073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8458438Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8458767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8459122Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8459533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8459944Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8460307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8460671Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8461030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8461438Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8461614Z 
2025-09-07T08:14:18.8461717Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8462066Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8462374Z     return mod(**inputs)
2025-09-07T08:14:18.8462711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8463110Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8463465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8463855Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8464202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8464577Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8464938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8465306Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8465659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8466031Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8466407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8466817Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8466990Z 
2025-09-07T08:14:18.8467091Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8467425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8467737Z     return mod(**inputs)
2025-09-07T08:14:18.8468070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8468436Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8468782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8469141Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8469475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8469830Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8470187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8470550Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8470919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8471286Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8471758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8472200Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8472412Z 
2025-09-07T08:14:18.8472506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8472905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8473264Z     return mod(**inputs)
2025-09-07T08:14:18.8473604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8473966Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8474321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8474683Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8475017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8475369Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8475722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8476091Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8476493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8476875Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8477236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8477722Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8477939Z 
2025-09-07T08:14:18.8478038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8478385Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8478699Z     return mod(**inputs)
2025-09-07T08:14:18.8479026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8479392Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8479744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8480120Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8480462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8480822Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8481194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8481567Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8481955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8482321Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8482685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8483050Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8483181Z 
2025-09-07T08:14:18.8483292Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8483646Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8483958Z     return mod(**inputs)
2025-09-07T08:14:18.8484300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8484667Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8485025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8485384Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8485727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8486078Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8486442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8486893Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8487246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8487621Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8487989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8488404Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8488563Z 
2025-09-07T08:14:18.8488667Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8489007Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8489322Z     return mod(**inputs)
2025-09-07T08:14:18.8489656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8490023Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8490413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8490783Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8491150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8491507Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8491868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8492231Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8492591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8492968Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8493345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8493756Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8493915Z 
2025-09-07T08:14:18.8494016Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8494373Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8494704Z     return mod(**inputs)
2025-09-07T08:14:18.8495053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8495419Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8495782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8496148Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8496497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8496860Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8497264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8497641Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8498008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8498382Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8498742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8499141Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8499302Z 
2025-09-07T08:14:18.8499398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8499750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8500122Z     return mod(**inputs)
2025-09-07T08:14:18.8500493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8500864Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8501219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8501585Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8501923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8502269Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8502633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8503005Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8503512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8503880Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8504297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8504694Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8504897Z 
2025-09-07T08:14:18.8504997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8505344Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8505665Z     return mod(**inputs)
2025-09-07T08:14:18.8506007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8506382Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8506746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8507115Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8507452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8507810Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8508178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8508556Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8508915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8509285Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8526412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8526819Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8526964Z 
2025-09-07T08:14:18.8527073Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8527445Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8527776Z     return mod(**inputs)
2025-09-07T08:14:18.8528142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8528567Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8528944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8529303Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8529648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8530009Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8530380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8530771Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8531259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8531714Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8532098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8532460Z     return self.weight * hidden_states
2025-09-07T08:14:18.8532586Z 
2025-09-07T08:14:18.8532691Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8533050Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8533380Z     return mod(**inputs)
2025-09-07T08:14:18.8533739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8534108Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8534464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8534869Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8535208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8535996Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8536364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8536740Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8537111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8537606Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8537998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8538362Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8538495Z 
2025-09-07T08:14:18.8538593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8538946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8539262Z     return mod(**inputs)
2025-09-07T08:14:18.8539598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8539959Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8540307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8540661Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8540991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8541336Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8541692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8542065Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8542429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8542825Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8543214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8543578Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8543707Z 
2025-09-07T08:14:18.8543801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8544140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8544476Z     return mod(**inputs)
2025-09-07T08:14:18.8544819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8545267Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8545623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8545978Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8546307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8546652Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8547017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8547405Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8547771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8548168Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8548606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8548973Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8549103Z 
2025-09-07T08:14:18.8549178Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8549432Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8549784Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8550095Z     return mod(**inputs)
2025-09-07T08:14:18.8550431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8550793Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8551141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8551495Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8551829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8552182Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8552533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8552898Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8553247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8553636Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8554018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8554373Z     return self.weight * hidden_states
2025-09-07T08:14:18.8554496Z 
2025-09-07T08:14:18.8554597Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8554937Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8555250Z     return mod(**inputs)
2025-09-07T08:14:18.8555574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8555932Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8556279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8556630Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8556955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8557299Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8557650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8558009Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8558403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8558803Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8559161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8559520Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8559646Z 
2025-09-07T08:14:18.8559737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8560074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8560379Z     return mod(**inputs)
2025-09-07T08:14:18.8560700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8561051Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8561397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8561786Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8562110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8562457Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8562838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8563216Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8563583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8563953Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8564307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8564663Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8564798Z 
2025-09-07T08:14:18.8564894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8565246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8565565Z     return mod(**inputs)
2025-09-07T08:14:18.8565887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8566245Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8566589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8566941Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8567265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8567607Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8567956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8568316Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8568665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8569029Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8569383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8569798Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8570001Z 
2025-09-07T08:14:18.8570096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8570435Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8570745Z     return mod(**inputs)
2025-09-07T08:14:18.8571090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8571501Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8571878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8572232Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8572558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8572908Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8573261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8573628Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8573976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8574335Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8574685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8575125Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8575297Z 
2025-09-07T08:14:18.8575391Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8575755Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8576066Z     return mod(**inputs)
2025-09-07T08:14:18.8576391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8576747Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8577088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8577502Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8577827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8578172Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8578522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8578881Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8579231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8579595Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8579949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8580353Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8580526Z 
2025-09-07T08:14:18.8580619Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8580973Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8581291Z     return mod(**inputs)
2025-09-07T08:14:18.8581621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8581982Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8582325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8582681Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8583002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8583348Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8583703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8584061Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8584416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8584835Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8585227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8585674Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8585881Z 
2025-09-07T08:14:18.8585974Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8586312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8586626Z     return mod(**inputs)
2025-09-07T08:14:18.8586949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8587313Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8587658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8588024Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8588384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8588741Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8589120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8589489Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8589837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8590199Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8590564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8590988Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8591200Z 
2025-09-07T08:14:18.8591301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8591637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8591949Z     return mod(**inputs)
2025-09-07T08:14:18.8592269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8592623Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8592972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8593322Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8593646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8593986Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8594341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8594711Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8595073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8595431Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8595787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8596145Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8596272Z 
2025-09-07T08:14:18.8596368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8596708Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8597008Z     return mod(**inputs)
2025-09-07T08:14:18.8597334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8597694Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8598082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8598492Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8598819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8599168Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8599515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8599880Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8600226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8600589Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8600943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8601343Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8601534Z 
2025-09-07T08:14:18.8601629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8601962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8602300Z     return mod(**inputs)
2025-09-07T08:14:18.8602622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8602994Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8603509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8603865Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8604199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8604543Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8604904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8605265Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8605616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8605983Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8606333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8606735Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8606886Z 
2025-09-07T08:14:18.8606976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8607312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8607623Z     return mod(**inputs)
2025-09-07T08:14:18.8607950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8608318Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8608661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8609028Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8609349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8609698Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8610044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8610409Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8610764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8611121Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8611556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8612000Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8612167Z 
2025-09-07T08:14:18.8612265Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8612607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8612927Z     return mod(**inputs)
2025-09-07T08:14:18.8613271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8613645Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8614002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8614369Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8614705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8615101Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8615459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8615875Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8616235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8616597Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8616967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8617425Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8617584Z 
2025-09-07T08:14:18.8617686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8618042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8618358Z     return mod(**inputs)
2025-09-07T08:14:18.8618699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8619062Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8619423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8619778Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8620112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8620471Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8620826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8621191Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8621543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8621913Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8622276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8622635Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8622761Z 
2025-09-07T08:14:18.8622849Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8623064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8623409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8623726Z     return mod(**inputs)
2025-09-07T08:14:18.8624060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8624426Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8624776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8625234Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8625563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8625926Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8626276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8626654Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8627026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8627419Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8627799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8628160Z     return self.weight * hidden_states
2025-09-07T08:14:18.8628289Z 
2025-09-07T08:14:18.8628427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8628768Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8629083Z     return mod(**inputs)
2025-09-07T08:14:18.8629440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8629819Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8630172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8630535Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8630863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8631207Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8631568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8631944Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8632315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8632718Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8633113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8633487Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8633616Z 
2025-09-07T08:14:18.8633715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8634070Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8634384Z     return mod(**inputs)
2025-09-07T08:14:18.8634715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8635091Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8635462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8635844Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8636180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8636541Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8636899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8637274Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8637651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8638054Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8638488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8638904Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8639049Z 
2025-09-07T08:14:18.8639152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8639497Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8639816Z     return mod(**inputs)
2025-09-07T08:14:18.8640145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8640523Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8640878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8641237Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8641571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8641960Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8642340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8642708Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8643131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8643529Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8643933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8644296Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8644421Z 
2025-09-07T08:14:18.8644493Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8644711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8645053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8645367Z     return mod(**inputs)
2025-09-07T08:14:18.8645691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8646051Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8646400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8646752Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8647076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8647418Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8647775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8648139Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8648495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8648890Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8649267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8649632Z     return self.weight * hidden_states
2025-09-07T08:14:18.8649763Z 
2025-09-07T08:14:18.8649856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8650211Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8650518Z     return mod(**inputs)
2025-09-07T08:14:18.8650852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8651215Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8651574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8652028Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8652355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8652708Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8653066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8653433Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8653791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8654174Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8654530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8654894Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8655037Z 
2025-09-07T08:14:18.8655138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8655511Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8655836Z     return mod(**inputs)
2025-09-07T08:14:18.8656201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8656580Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8656929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8657320Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8657659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8658011Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8658383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8658755Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8659115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8659489Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8659853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8660219Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8660345Z 
2025-09-07T08:14:18.8660438Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8660773Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8661085Z     return mod(**inputs)
2025-09-07T08:14:18.8661426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8661789Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8662154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8662513Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8662843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8663200Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8663556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8663931Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8664281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8664650Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8665013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8665511Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8665696Z 
2025-09-07T08:14:18.8665787Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8666132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8666451Z     return mod(**inputs)
2025-09-07T08:14:18.8666797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8667160Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8667508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8667880Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8668205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8668548Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8668955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8669322Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8669729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8670101Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8670456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8670864Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8671061Z 
2025-09-07T08:14:18.8671154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8671511Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8671819Z     return mod(**inputs)
2025-09-07T08:14:18.8672165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8672533Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8672877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8673254Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8673575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8673972Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8674329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8674690Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8675061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8675424Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8675803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8676227Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8676401Z 
2025-09-07T08:14:18.8676499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8676831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8677147Z     return mod(**inputs)
2025-09-07T08:14:18.8677476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8677845Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8678188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8678632Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8678972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8679375Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8679729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8680107Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8680456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8680817Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8681172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8681632Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8681835Z 
2025-09-07T08:14:18.8681925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8682317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8682669Z     return mod(**inputs)
2025-09-07T08:14:18.8682997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8683399Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8683750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8684108Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8684445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8684790Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8685142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8685509Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8685879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8686243Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8686603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8687039Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8687246Z 
2025-09-07T08:14:18.8687336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8687674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8687988Z     return mod(**inputs)
2025-09-07T08:14:18.8688312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8688684Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8689043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8689401Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8689745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8690108Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8690469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8690847Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8691206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8691580Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8691926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8692327Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8692504Z 
2025-09-07T08:14:18.8692596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8692936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8693245Z     return mod(**inputs)
2025-09-07T08:14:18.8693572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8693931Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8694282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8694645Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8694970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8695319Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8695709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8696086Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8696491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8696863Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8697301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8697701Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8697859Z 
2025-09-07T08:14:18.8697963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8698304Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8698612Z     return mod(**inputs)
2025-09-07T08:14:18.8698939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8699316Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8699666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8700021Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8700349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8700705Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8701061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8701430Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8701782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8702168Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8702530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8702918Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8703205Z 
2025-09-07T08:14:18.8703303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8703639Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8703955Z     return mod(**inputs)
2025-09-07T08:14:18.8704284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8704648Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8704987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8705353Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8705684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8706149Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8706526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8706894Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8707266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8707634Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8708013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8708401Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8708566Z 
2025-09-07T08:14:18.8708665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8709010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8709340Z     return mod(**inputs)
2025-09-07T08:14:18.8709735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8710107Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8710512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8710883Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8711241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8711599Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8711961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8712331Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8712691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8713060Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8713415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8713812Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8713971Z 
2025-09-07T08:14:18.8714068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8714413Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8714720Z     return mod(**inputs)
2025-09-07T08:14:18.8715045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8715401Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8715745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8716115Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8716439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8716787Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8717146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8717512Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8717875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8718235Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8718588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8718947Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8719116Z 
2025-09-07T08:14:18.8719216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8719597Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8719902Z     return mod(**inputs)
2025-09-07T08:14:18.8720239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8720600Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8720946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8721301Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8721629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8721981Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8722340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8722712Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8723103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.8723528Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.8723751Z 
2025-09-07T08:14:18.8723829Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8724054Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8724386Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8724700Z     return mod(**inputs)
2025-09-07T08:14:18.8725037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8725395Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8725744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8726105Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8726431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8726777Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8727140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8727517Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8727885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8728271Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8728648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8729008Z     return self.weight * hidden_states
2025-09-07T08:14:18.8729135Z 
2025-09-07T08:14:18.8729228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8729566Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8729875Z     return mod(**inputs)
2025-09-07T08:14:18.8730203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8730562Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8730904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8731267Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8731599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8731955Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8732311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8732785Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8733162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8733567Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8733971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8734332Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8734465Z 
2025-09-07T08:14:18.8734561Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8734910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8735224Z     return mod(**inputs)
2025-09-07T08:14:18.8735558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8735924Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8736323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8736691Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8737061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8737488Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8737856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8738233Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8738604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8739011Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8739404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8739775Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8739911Z 
2025-09-07T08:14:18.8740005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8740358Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8740667Z     return mod(**inputs)
2025-09-07T08:14:18.8740995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8741359Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8741712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8742079Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8742406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8742787Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8743171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8743554Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8743926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8744321Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8744718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8745086Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8745215Z 
2025-09-07T08:14:18.8745297Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8745529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8745871Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8746264Z     return mod(**inputs)
2025-09-07T08:14:18.8746613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8746982Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8747333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8747697Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8748038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8748402Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8748764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8749126Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8749491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8749939Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8750326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8750719Z     return self.weight * hidden_states
2025-09-07T08:14:18.8750856Z 
2025-09-07T08:14:18.8750950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8751299Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8751620Z     return mod(**inputs)
2025-09-07T08:14:18.8751948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8752306Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8752666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8753031Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8753372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8753735Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8754099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8754470Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8754835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8755207Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8755560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8755931Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8756063Z 
2025-09-07T08:14:18.8756157Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8756502Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8756819Z     return mod(**inputs)
2025-09-07T08:14:18.8757141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8757498Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8757847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8758202Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8758527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8758877Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8759229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8759652Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8760059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8760419Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8760788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8761148Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8761276Z 
2025-09-07T08:14:18.8761377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8761727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8762036Z     return mod(**inputs)
2025-09-07T08:14:18.8762372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8762737Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8763125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8763493Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8763827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8764214Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8764576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8764950Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8765303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8765669Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8766027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8766441Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8766622Z 
2025-09-07T08:14:18.8766720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8767055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8767370Z     return mod(**inputs)
2025-09-07T08:14:18.8767699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8768056Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8768398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8768757Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8769084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8769429Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8769790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8770155Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8770514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8770881Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8771251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8771656Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8771837Z 
2025-09-07T08:14:18.8771929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8772273Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8772579Z     return mod(**inputs)
2025-09-07T08:14:18.8772962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8773354Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8773697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8774065Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8774398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8774746Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8775106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8775467Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8775819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8776198Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8776583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8776991Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8777212Z 
2025-09-07T08:14:18.8777338Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8777684Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8777988Z     return mod(**inputs)
2025-09-07T08:14:18.8778323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8790225Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8790648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8791028Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8791377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8791743Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8792115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8792491Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8792854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8793219Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8793578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8794016Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8794221Z 
2025-09-07T08:14:18.8794328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8794685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8795012Z     return mod(**inputs)
2025-09-07T08:14:18.8795374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8795746Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8796097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8796456Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8796798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8797151Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8797510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8797874Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8798358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8798766Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8799126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8799562Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8799766Z 
2025-09-07T08:14:18.8799866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8800214Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8800526Z     return mod(**inputs)
2025-09-07T08:14:18.8800855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8801216Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8801563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8801956Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8802287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8802678Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8803035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8803596Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8803955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8804314Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8804672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8805045Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8805177Z 
2025-09-07T08:14:18.8805276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8805615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8805927Z     return mod(**inputs)
2025-09-07T08:14:18.8806255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8806608Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8806961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8807324Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8807662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8808013Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8808370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8808738Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8809091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8809456Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8809810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8810201Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8810360Z 
2025-09-07T08:14:18.8810457Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8810800Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8811111Z     return mod(**inputs)
2025-09-07T08:14:18.8811439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8811959Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8812310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8812662Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8812991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8813338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8813688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8814047Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8814397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8814751Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8815104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8815542Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8815694Z 
2025-09-07T08:14:18.8815795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8816178Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8816488Z     return mod(**inputs)
2025-09-07T08:14:18.8816818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8817226Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8817577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8817930Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8818259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8818611Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8818972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8819332Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8819683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8820047Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8820400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8820784Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8820933Z 
2025-09-07T08:14:18.8821030Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8821362Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8821671Z     return mod(**inputs)
2025-09-07T08:14:18.8821998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8822354Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8822695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8823052Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8823377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8823722Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8824075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8824436Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8824790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8825238Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8825599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8825985Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8826141Z 
2025-09-07T08:14:18.8826237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8826577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8826880Z     return mod(**inputs)
2025-09-07T08:14:18.8827207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8827559Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8828478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8828836Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8829203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8829554Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8829939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8830302Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8830656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8831013Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8831362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8831714Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8831837Z 
2025-09-07T08:14:18.8831909Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8832127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8832465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8832776Z     return mod(**inputs)
2025-09-07T08:14:18.8833102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8833169Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8833400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8833462Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8833680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8833749Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8833978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8834062Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8834293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8834384Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8834609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8834682Z     return self.weight * hidden_states
2025-09-07T08:14:18.8834685Z 
2025-09-07T08:14:18.8834778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8834967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8835025Z     return mod(**inputs)
2025-09-07T08:14:18.8835256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8835390Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8835656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8835722Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8835935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8836004Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8836234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8836311Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8836544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8836653Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8836882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8836992Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8836995Z 
2025-09-07T08:14:18.8837085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8837305Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8837363Z     return mod(**inputs)
2025-09-07T08:14:18.8837602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8837667Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8837898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8837962Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8838172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8838244Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8838470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8838548Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8838777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8838876Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8839105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8839179Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8839182Z 
2025-09-07T08:14:18.8839274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8839457Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8839514Z     return mod(**inputs)
2025-09-07T08:14:18.8839751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8839813Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8840059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8840122Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8840346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8840417Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8840652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8840732Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8840960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8841103Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8841366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8841437Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8841441Z 
2025-09-07T08:14:18.8841518Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8841612Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8841810Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8841868Z     return mod(**inputs)
2025-09-07T08:14:18.8842101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8842169Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8842398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8842462Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8842718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8842788Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8843057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8843130Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8843360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8843459Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8843688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8843755Z     return self.weight * hidden_states
2025-09-07T08:14:18.8843758Z 
2025-09-07T08:14:18.8843851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8844042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8844097Z     return mod(**inputs)
2025-09-07T08:14:18.8844331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8844394Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8844627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8844692Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8844905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8844973Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8845198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8845270Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8845502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8845576Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8845807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8845874Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8845878Z 
2025-09-07T08:14:18.8845970Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8846153Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8846207Z     return mod(**inputs)
2025-09-07T08:14:18.8846439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8846502Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8846809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8846900Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8847111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8847185Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8847410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8847482Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8847706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8847776Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8848005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8848074Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8848079Z 
2025-09-07T08:14:18.8848206Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8848392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8848452Z     return mod(**inputs)
2025-09-07T08:14:18.8848712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8848776Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8849008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8849070Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8849288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8849354Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8849584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8849670Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8849895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8849968Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8850195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8850316Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8850322Z 
2025-09-07T08:14:18.8850410Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8850592Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8850650Z     return mod(**inputs)
2025-09-07T08:14:18.8850879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8850948Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8851178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8851240Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8851454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8851520Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8851748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8851816Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8852039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8852109Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8852375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8852524Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8852527Z 
2025-09-07T08:14:18.8852617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8852803Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8852859Z     return mod(**inputs)
2025-09-07T08:14:18.8853088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8853151Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8853376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8853441Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8853652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8853755Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8853997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8854105Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8854341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8854412Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8854636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8854753Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8854756Z 
2025-09-07T08:14:18.8854846Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8855034Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8855096Z     return mod(**inputs)
2025-09-07T08:14:18.8855330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8855394Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8855623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8855686Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8855896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8855967Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8856194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8856264Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8856492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8856563Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8856790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8856936Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8856939Z 
2025-09-07T08:14:18.8857033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8857264Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8860319Z     return mod(**inputs)
2025-09-07T08:14:18.8860597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8860671Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8860907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8861307Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8861529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8861608Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8861846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8861920Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8862152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8862249Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8862478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8862626Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8862632Z 
2025-09-07T08:14:18.8862767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8862971Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8863030Z     return mod(**inputs)
2025-09-07T08:14:18.8863346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8863414Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8863645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8863714Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8863930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8864010Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8864238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8864317Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8864545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8864618Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8864852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8864926Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8864930Z 
2025-09-07T08:14:18.8865031Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8865229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8865290Z     return mod(**inputs)
2025-09-07T08:14:18.8865529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8865594Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8865832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8865896Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8866111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8866187Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8866418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8866497Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8866802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8866882Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8867108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8867253Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8867256Z 
2025-09-07T08:14:18.8867359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8867553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8867620Z     return mod(**inputs)
2025-09-07T08:14:18.8867854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8867919Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8868160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8868224Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8868443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8868514Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8868779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8868857Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8869130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8869211Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8869437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8869537Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8869540Z 
2025-09-07T08:14:18.8869631Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8869819Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8869878Z     return mod(**inputs)
2025-09-07T08:14:18.8870111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8870177Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8870407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8870470Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8870687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8870755Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8870987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8871059Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8871289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8871360Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8871586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8871684Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8871688Z 
2025-09-07T08:14:18.8871778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8871960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8872015Z     return mod(**inputs)
2025-09-07T08:14:18.8872245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8872359Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8872590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8872655Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8872907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8872974Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8873210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8873280Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8873510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8873580Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8873809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8873903Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8873907Z 
2025-09-07T08:14:18.8873995Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8874211Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8874271Z     return mod(**inputs)
2025-09-07T08:14:18.8874505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8874598Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8874830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8874895Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8875107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8875177Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8875403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8875471Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8875705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8875773Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8876002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8876070Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8876074Z 
2025-09-07T08:14:18.8876164Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8876345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8876402Z     return mod(**inputs)
2025-09-07T08:14:18.8876633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8876693Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8876923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8876986Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8877196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8877265Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8877489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8877560Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8877784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.8877949Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.8877952Z 
2025-09-07T08:14:18.8878024Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8878111Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8878295Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8878393Z     return mod(**inputs)
2025-09-07T08:14:18.8878625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8878689Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8878918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8878980Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8879190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8879259Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8879485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8879567Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8879828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8879915Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8880176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8880245Z     return self.weight * hidden_states
2025-09-07T08:14:18.8880248Z 
2025-09-07T08:14:18.8880339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8880522Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8880580Z     return mod(**inputs)
2025-09-07T08:14:18.8880811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8880872Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8881105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8881168Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8881379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8881449Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8881675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8881755Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8881979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8882090Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8882318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8882387Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8882392Z 
2025-09-07T08:14:18.8882483Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8882663Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8882723Z     return mod(**inputs)
2025-09-07T08:14:18.8882951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8883012Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8883242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8883341Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8883553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8883618Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8883843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8883953Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8884180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8884286Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8884513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8884585Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8884592Z 
2025-09-07T08:14:18.8884681Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8884861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8884919Z     return mod(**inputs)
2025-09-07T08:14:18.8885148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8885254Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8885487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8885583Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8885800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8885866Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8886095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8886171Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8886397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8886498Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8886726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8886796Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8886800Z 
2025-09-07T08:14:18.8886869Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8886961Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8887143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8887198Z     return mod(**inputs)
2025-09-07T08:14:18.8887430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8887494Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8887724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8887783Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8887996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8888066Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8888292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8888366Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8888593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8888688Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8888974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8889041Z     return self.weight * hidden_states
2025-09-07T08:14:18.8889044Z 
2025-09-07T08:14:18.8889134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8889317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8889411Z     return mod(**inputs)
2025-09-07T08:14:18.8889641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8889706Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8889935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8889996Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8890213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8890279Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8890504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8890578Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8890865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8890943Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8891203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8891273Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8891281Z 
2025-09-07T08:14:18.8891368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8891548Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8891608Z     return mod(**inputs)
2025-09-07T08:14:18.8891837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8891902Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8892130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8892195Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8892408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8892477Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8892708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8892775Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8893002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8893077Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8893302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8893374Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8893377Z 
2025-09-07T08:14:18.8893468Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8893652Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8893708Z     return mod(**inputs)
2025-09-07T08:14:18.8893937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8894002Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8894230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8894296Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8894565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8894634Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8894865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8894970Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8895201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8895272Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8895498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8895623Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8895627Z 
2025-09-07T08:14:18.8895716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8895905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8895960Z     return mod(**inputs)
2025-09-07T08:14:18.8896191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8896256Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8896516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8896582Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8896827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8896896Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8897123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8897251Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8897487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8897556Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8897784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8897896Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8897899Z 
2025-09-07T08:14:18.8897987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8898172Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8898227Z     return mod(**inputs)
2025-09-07T08:14:18.8898457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8898518Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8898751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8898810Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8899020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8899090Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8899318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8899386Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8899612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8899682Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8899906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8900062Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8900065Z 
2025-09-07T08:14:18.8900155Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8900335Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8900394Z     return mod(**inputs)
2025-09-07T08:14:18.8900665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8900730Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8900963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8901024Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8901238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8901305Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8901532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8901608Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8901833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8901941Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8902169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8902351Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8902358Z 
2025-09-07T08:14:18.8902449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8902634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8902692Z     return mod(**inputs)
2025-09-07T08:14:18.8902921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8902988Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8903443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8903505Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8903723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8903790Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8904024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8904095Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8904321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8904398Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8904624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8904767Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8904771Z 
2025-09-07T08:14:18.8904862Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8905053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8905110Z     return mod(**inputs)
2025-09-07T08:14:18.8905341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8905405Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8905634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8905698Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8906000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8906071Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8906320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8906439Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8906675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8906747Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8906975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8907052Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8907056Z 
2025-09-07T08:14:18.8907151Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8907345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8907405Z     return mod(**inputs)
2025-09-07T08:14:18.8907645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8907711Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8907988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8908056Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8908317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8908396Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8908626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8908697Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8908932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8909004Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8909237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8909338Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8909342Z 
2025-09-07T08:14:18.8909440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8909640Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8909700Z     return mod(**inputs)
2025-09-07T08:14:18.8909935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8910001Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8910236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8910303Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8910517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8910590Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8910825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8910903Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8911131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8911203Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8911436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8911531Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8911571Z 
2025-09-07T08:14:18.8911665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8911849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8911911Z     return mod(**inputs)
2025-09-07T08:14:18.8912145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8912259Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8912491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8912555Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8912773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8912844Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8913070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8913147Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8913375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8913447Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8913710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8913804Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8913811Z 
2025-09-07T08:14:18.8913931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8914117Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8914181Z     return mod(**inputs)
2025-09-07T08:14:18.8914413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8914485Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8914714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8914778Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8914996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8915063Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8915296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8915369Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8915594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8915669Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8915896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8915992Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8915996Z 
2025-09-07T08:14:18.8916084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8916270Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8916335Z     return mod(**inputs)
2025-09-07T08:14:18.8916563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8916630Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8916861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8916929Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8917139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8917243Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8917477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8917546Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8917781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8917882Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8918113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8918189Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8918193Z 
2025-09-07T08:14:18.8918266Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8918373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8918555Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8918614Z     return mod(**inputs)
2025-09-07T08:14:18.8918849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8918913Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8919181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8919244Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8919463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8919561Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8919792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8919877Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8920106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8920202Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8920430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8920498Z     return self.weight * hidden_states
2025-09-07T08:14:18.8920503Z 
2025-09-07T08:14:18.8920603Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8920786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8920847Z     return mod(**inputs)
2025-09-07T08:14:18.8921078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8921142Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8921378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8921443Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8921659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8921727Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8921959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8922039Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8922265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8922383Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8922608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8922683Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8922686Z 
2025-09-07T08:14:18.8922812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8922994Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8923059Z     return mod(**inputs)
2025-09-07T08:14:18.8923290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8923391Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8923623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8923691Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8923902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8923970Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8924207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8924285Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8924516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8924614Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8924872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8924953Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8924955Z 
2025-09-07T08:14:18.8925045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8925261Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8925319Z     return mod(**inputs)
2025-09-07T08:14:18.8925549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8925618Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8925851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8925917Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8926135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8926218Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8926454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8926536Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8926770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8926879Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8927110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8927183Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8927186Z 
2025-09-07T08:14:18.8927259Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8927362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8927557Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8927623Z     return mod(**inputs)
2025-09-07T08:14:18.8927857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8927926Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8928163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8928225Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8928448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8928556Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8928791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8928862Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8929093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8929230Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8929460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8929531Z     return self.weight * hidden_states
2025-09-07T08:14:18.8929535Z 
2025-09-07T08:14:18.8929629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8929815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8929880Z     return mod(**inputs)
2025-09-07T08:14:18.8930109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8930177Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8930408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8930508Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8930732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8930830Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8931064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8931134Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8931364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8931438Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8931662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8931732Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8931735Z 
2025-09-07T08:14:18.8931831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8932018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8932073Z     return mod(**inputs)
2025-09-07T08:14:18.8932304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8932367Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8932597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8932662Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8932875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8932944Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8933171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8933244Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8933478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8933551Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8933779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8933845Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8933848Z 
2025-09-07T08:14:18.8933937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8934188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8934244Z     return mod(**inputs)
2025-09-07T08:14:18.8934475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8934536Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8934803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8934868Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8935081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8935148Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8935373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8935442Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8935669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8935738Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8935964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8936123Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8936127Z 
2025-09-07T08:14:18.8936218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8936431Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8936487Z     return mod(**inputs)
2025-09-07T08:14:18.8936718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8936779Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8937011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8937073Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8937326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8937394Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8937624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8937695Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8937920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8937991Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8938215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8938325Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8938330Z 
2025-09-07T08:14:18.8938421Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8938602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8938660Z     return mod(**inputs)
2025-09-07T08:14:18.8938891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8938952Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8939186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8939247Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8939460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8939524Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8939808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8939879Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8940104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8940213Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8940441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8940553Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8940558Z 
2025-09-07T08:14:18.8940648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8940829Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8940887Z     return mod(**inputs)
2025-09-07T08:14:18.8941115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8941184Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8941412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8941473Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8941730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8941801Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8942063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8942132Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8942361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8942430Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8942655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8942803Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8942807Z 
2025-09-07T08:14:18.8942897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8943085Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8943139Z     return mod(**inputs)
2025-09-07T08:14:18.8943369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8943433Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8943659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8943722Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8943933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8943999Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8944227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8944298Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8944529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8944596Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8944825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8944959Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8944962Z 
2025-09-07T08:14:18.8945048Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8945232Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8945345Z     return mod(**inputs)
2025-09-07T08:14:18.8945578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8945639Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8945912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8945976Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8946190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8946258Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8946482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8946557Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8946784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8946851Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8947079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8947149Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8947185Z 
2025-09-07T08:14:18.8947279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8947460Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8947546Z     return mod(**inputs)
2025-09-07T08:14:18.8947786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8947849Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8948081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8948142Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8948357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8948427Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8948654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8948728Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8948954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8949027Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8949253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8949353Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8949358Z 
2025-09-07T08:14:18.8949454Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8949635Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8949695Z     return mod(**inputs)
2025-09-07T08:14:18.8949925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8949988Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8950215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8950278Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8950492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8950557Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8950784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8950893Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8951121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8951194Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8951458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8951552Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8951555Z 
2025-09-07T08:14:18.8951653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8951842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8951902Z     return mod(**inputs)
2025-09-07T08:14:18.8952143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8952212Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8952441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8952504Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8952723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8952857Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8953094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8953214Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8953446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8953519Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8953749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8953857Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8953861Z 
2025-09-07T08:14:18.8953957Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8954148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8954211Z     return mod(**inputs)
2025-09-07T08:14:18.8954446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8954518Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8954750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8954832Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8955047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8955120Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8955354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8955427Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8955661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8955736Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8955970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8956067Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8956070Z 
2025-09-07T08:14:18.8956170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8956363Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8956423Z     return mod(**inputs)
2025-09-07T08:14:18.8956720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8956784Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8957017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8957129Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8957342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8957418Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8957648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8957730Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8957958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8958032Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8958264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.8958334Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.8958337Z 
2025-09-07T08:14:18.8958434Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8958653Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8958715Z     return mod(**inputs)
2025-09-07T08:14:18.8958990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8959055Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8959292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8959356Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8959571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8959651Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8959880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8959961Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8960188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.8960325Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.8960328Z 
2025-09-07T08:14:18.8960403Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8960493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8960683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8960741Z     return mod(**inputs)
2025-09-07T08:14:18.8960976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8961037Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8961265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8961337Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8961548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8961622Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8961853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8961938Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8962171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.8962295Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8962529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8962599Z     return self.weight * hidden_states
2025-09-07T08:14:18.8962602Z 
2025-09-07T08:14:18.8962735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8962918Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8962981Z     return mod(**inputs)
2025-09-07T08:14:18.8963219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8963282Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8963514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8963577Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8963791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8963865Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8964093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8964214Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8964444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8964594Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8964832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.8964903Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.8964907Z 
2025-09-07T08:14:18.8965005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8965192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8965255Z     return mod(**inputs)
2025-09-07T08:14:18.8965488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8965553Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8965794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8965857Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8966082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8966148Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8966375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8966459Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8966686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8966791Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8967018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.8967099Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.8967102Z 
2025-09-07T08:14:18.8967192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8967375Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8967444Z     return mod(**inputs)
2025-09-07T08:14:18.8967676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8967749Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8968017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8968080Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8968296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8968363Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8968636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.8968714Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.8968946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.8969057Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.8969283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.8969357Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.8969360Z 
2025-09-07T08:14:18.8969431Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.8969526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8969705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8969765Z     return mod(**inputs)
2025-09-07T08:14:18.8970033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8970099Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8970374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8970439Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8970652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8970729Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8970953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8971033Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8971262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.8971366Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.8971602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.8971670Z     return self.weight * hidden_states
2025-09-07T08:14:18.8971673Z 
2025-09-07T08:14:18.8971765Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8971947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8972004Z     return mod(**inputs)
2025-09-07T08:14:18.8972233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8972295Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8972525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8972587Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8972802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8972869Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8973096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8973172Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8973399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8973515Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8973743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.8973813Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.8973821Z 
2025-09-07T08:14:18.8973913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8974132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8974201Z     return mod(**inputs)
2025-09-07T08:14:18.8974430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8974497Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8974727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8974789Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8975007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8975072Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8975307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8975377Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8975654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8975731Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8975990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.8976062Z     key_states = self.k(current_states)
2025-09-07T08:14:18.8976065Z 
2025-09-07T08:14:18.8976155Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8976343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8976402Z     return mod(**inputs)
2025-09-07T08:14:18.8976632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8976703Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8976934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8977002Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8977270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8977338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8977569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8977640Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8977876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8977947Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8978172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8978295Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8978301Z 
2025-09-07T08:14:18.8978390Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8978573Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8978629Z     return mod(**inputs)
2025-09-07T08:14:18.8978861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8978922Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8979150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8979250Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8979461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8979529Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8979755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8979860Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8980092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8980163Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8980392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8980505Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8980511Z 
2025-09-07T08:14:18.8980614Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8980801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8980859Z     return mod(**inputs)
2025-09-07T08:14:18.8981127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8981194Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8981428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8981521Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8981739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8981813Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8982042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8982120Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8982349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8982421Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8982662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.8982775Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.8982778Z 
2025-09-07T08:14:18.8982876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8983055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8983112Z     return mod(**inputs)
2025-09-07T08:14:18.8983342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8983407Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8983640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8983700Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8983921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8983991Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8984216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8984294Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8984522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8984598Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8984826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8985009Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8985016Z 
2025-09-07T08:14:18.8985105Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8985288Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8985386Z     return mod(**inputs)
2025-09-07T08:14:18.8985617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8985684Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8985915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8985974Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8986189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8986259Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8986497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8986566Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8986824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8986901Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8987160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.8987306Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.8987310Z 
2025-09-07T08:14:18.8987398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8987589Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8987650Z     return mod(**inputs)
2025-09-07T08:14:18.8987883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8987953Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8988183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8988249Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8988461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8988530Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8988763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8988832Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8989066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8989138Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8989367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.8989447Z     value_states = self.v(current_states)
2025-09-07T08:14:18.8989452Z 
2025-09-07T08:14:18.8989545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8989734Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8989793Z     return mod(**inputs)
2025-09-07T08:14:18.8990030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8990095Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8990324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8990423Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8990634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8990707Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8990933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8991039Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8991277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8991354Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8991588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8991691Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8991694Z 
2025-09-07T08:14:18.8991795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8991984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8992044Z     return mod(**inputs)
2025-09-07T08:14:18.8992281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8992383Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8992624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8992688Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8992932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8993007Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8993236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8993314Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8993540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8993609Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8993839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8993932Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8993936Z 
2025-09-07T08:14:18.8994028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8994212Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8994269Z     return mod(**inputs)
2025-09-07T08:14:18.8994497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8994559Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8994793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8994855Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8995070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8995140Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8995366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8995443Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8995673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8995747Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8995976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.8996124Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.8996132Z 
2025-09-07T08:14:18.8996225Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8996409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8996472Z     return mod(**inputs)
2025-09-07T08:14:18.8996762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8996836Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8997068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8997131Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8997350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8997415Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8997650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8997720Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.8997945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.8998078Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.8998306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.8998432Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.8998436Z 
2025-09-07T08:14:18.8998525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.8998713Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.8998770Z     return mod(**inputs)
2025-09-07T08:14:18.8998999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.8999069Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.8999299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.8999369Z     layer_outputs = layer_module(
2025-09-07T08:14:18.8999586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.8999654Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.8999891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.8999962Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9000197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9000268Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9000496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9000572Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9000575Z 
2025-09-07T08:14:18.9000647Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9000745Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9000931Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9000988Z     return mod(**inputs)
2025-09-07T08:14:18.9001228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9001294Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9001532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9001595Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9001848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9001916Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9002145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9002267Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9002497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9002595Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9002822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9002894Z     return self.weight * hidden_states
2025-09-07T08:14:18.9002897Z 
2025-09-07T08:14:18.9002997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9003343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9003403Z     return mod(**inputs)
2025-09-07T08:14:18.9003633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9003698Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9004024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9004090Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9004368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9004437Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9004668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9004747Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9004974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9005089Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9005318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9005398Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9005402Z 
2025-09-07T08:14:18.9005495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9005685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9005753Z     return mod(**inputs)
2025-09-07T08:14:18.9005986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9006061Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9006291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9006359Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9006573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9006641Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9006877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9006955Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9007193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9007296Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9007525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9007660Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9007663Z 
2025-09-07T08:14:18.9007754Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9007944Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9008002Z     return mod(**inputs)
2025-09-07T08:14:18.9008287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9008358Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9008592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9008662Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9008875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9008945Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9009173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9009252Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9009488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9009621Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9009863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9009934Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9009966Z 
2025-09-07T08:14:18.9010042Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9010139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9010322Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9010388Z     return mod(**inputs)
2025-09-07T08:14:18.9010621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9010685Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9010924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9010987Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9011209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9011276Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9011515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9011589Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9011815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9011926Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9012154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9012229Z     return self.weight * hidden_states
2025-09-07T08:14:18.9012232Z 
2025-09-07T08:14:18.9012319Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9012505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9012567Z     return mod(**inputs)
2025-09-07T08:14:18.9012800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9012868Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9013097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9013168Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9019288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9019371Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9019644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9019805Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9020065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9020146Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9020393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9020469Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9020474Z 
2025-09-07T08:14:18.9020581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9020796Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9020862Z     return mod(**inputs)
2025-09-07T08:14:18.9021111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9021184Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9021455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9021528Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9021786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9021861Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9022102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9022176Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9022409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9022487Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9022720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9022788Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9022794Z 
2025-09-07T08:14:18.9022894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9023093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9023154Z     return mod(**inputs)
2025-09-07T08:14:18.9023390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9023455Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9023687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9023756Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9023970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9024045Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9024272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9024347Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9024596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9024675Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9024915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9025042Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9025099Z 
2025-09-07T08:14:18.9025205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9025404Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9025464Z     return mod(**inputs)
2025-09-07T08:14:18.9025710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9025814Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9026053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9026116Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9026335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9026412Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9026645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9026727Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9026954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9027033Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9027295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9027411Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9027415Z 
2025-09-07T08:14:18.9027538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9027734Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9027797Z     return mod(**inputs)
2025-09-07T08:14:18.9028028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9028096Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9028331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9028394Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9028612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9028682Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9028908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9028987Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9029214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9029295Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9029520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9029639Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9029642Z 
2025-09-07T08:14:18.9029735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9029921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9029989Z     return mod(**inputs)
2025-09-07T08:14:18.9030220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9030287Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9030520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9030583Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9030802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9030906Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9031140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9031211Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9031445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9031551Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9031777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9031933Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9031937Z 
2025-09-07T08:14:18.9032029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9032218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9032277Z     return mod(**inputs)
2025-09-07T08:14:18.9032520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9032590Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9032823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9032925Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9033143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9033256Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9033503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9033583Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9033818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9033897Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9034132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9034279Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9034284Z 
2025-09-07T08:14:18.9034385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9034586Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9034644Z     return mod(**inputs)
2025-09-07T08:14:18.9034886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9034951Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9035182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9035251Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9035465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9035537Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9035767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9035841Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9036066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9036139Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9036366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9036434Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9036438Z 
2025-09-07T08:14:18.9036530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9036754Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9036810Z     return mod(**inputs)
2025-09-07T08:14:18.9037042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9037142Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9037379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9037439Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9037654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9037725Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9037951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9038023Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9038247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9038321Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9038579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9038682Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9038685Z 
2025-09-07T08:14:18.9038783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9039001Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9039062Z     return mod(**inputs)
2025-09-07T08:14:18.9039296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9039359Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9039596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9039658Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9039873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9039942Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9040170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9040244Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9040472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9040547Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9040804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9040909Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9040912Z 
2025-09-07T08:14:18.9041004Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9041188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9041248Z     return mod(**inputs)
2025-09-07T08:14:18.9041479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9041542Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9041772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9041831Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9042043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9042108Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9042379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9042447Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9042673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9042783Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9043010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9043106Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9043110Z 
2025-09-07T08:14:18.9043198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9043383Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9043438Z     return mod(**inputs)
2025-09-07T08:14:18.9043667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9043731Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9043960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9044025Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9044272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9044337Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9044612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9044682Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9044916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9044986Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9045214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9045310Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9045313Z 
2025-09-07T08:14:18.9045403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9045593Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9045648Z     return mod(**inputs)
2025-09-07T08:14:18.9045878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9045938Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9046167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9046228Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9046442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9046515Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9046742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9046810Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9047046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9047115Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9047354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9047421Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9047424Z 
2025-09-07T08:14:18.9047515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9047697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9047788Z     return mod(**inputs)
2025-09-07T08:14:18.9048025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9048087Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9048321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9048443Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9048655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9048723Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9048949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9049019Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9049251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.9049377Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9049385Z 
2025-09-07T08:14:18.9049458Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9049549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9049772Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9049830Z     return mod(**inputs)
2025-09-07T08:14:18.9050099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9050162Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9050391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9050452Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9050662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9050730Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9050955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9051037Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9051267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9051352Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9051580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9051650Z     return self.weight * hidden_states
2025-09-07T08:14:18.9051654Z 
2025-09-07T08:14:18.9051750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9051939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9051998Z     return mod(**inputs)
2025-09-07T08:14:18.9052231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9052293Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9052526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9052589Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9052806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9052876Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9053102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9053184Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9053409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9053559Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9053790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9053897Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9053902Z 
2025-09-07T08:14:18.9054004Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9054193Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9054260Z     return mod(**inputs)
2025-09-07T08:14:18.9054492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9054557Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9054794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9054868Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9055089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9055158Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9055422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9055509Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9055770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9055879Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9056106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9056180Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9056191Z 
2025-09-07T08:14:18.9056283Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9056468Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9056532Z     return mod(**inputs)
2025-09-07T08:14:18.9056763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9056835Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9057065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9057131Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9057409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9057482Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9057725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9057811Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9058040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9058146Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9058374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9058445Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9058448Z 
2025-09-07T08:14:18.9058521Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9058614Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9058799Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9058855Z     return mod(**inputs)
2025-09-07T08:14:18.9059087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9059188Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9059419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9059481Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9059733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9059804Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9060045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9060116Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9060342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9060438Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9060665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9060732Z     return self.weight * hidden_states
2025-09-07T08:14:18.9060735Z 
2025-09-07T08:14:18.9060826Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9061040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9061099Z     return mod(**inputs)
2025-09-07T08:14:18.9061330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9061422Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9061665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9061729Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9061945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9062012Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9062236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9062308Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9062534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9062608Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9062836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9062904Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9062911Z 
2025-09-07T08:14:18.9062999Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9063179Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9063238Z     return mod(**inputs)
2025-09-07T08:14:18.9063465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9063529Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9063757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9063819Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9064031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9064097Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9064324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9064391Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9064621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9064728Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9064952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9065021Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9065063Z 
2025-09-07T08:14:18.9065152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9065334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9065391Z     return mod(**inputs)
2025-09-07T08:14:18.9065620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9065685Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9065913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9065975Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9066185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9066249Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9066478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9066595Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9066835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9066942Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9067180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9067309Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9067313Z 
2025-09-07T08:14:18.9067405Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9067599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9067655Z     return mod(**inputs)
2025-09-07T08:14:18.9067892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9067958Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9068191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9068255Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9068468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9068540Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9068765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9068837Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9069070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9069142Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9069378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9069494Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9069497Z 
2025-09-07T08:14:18.9069586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9069777Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9069832Z     return mod(**inputs)
2025-09-07T08:14:18.9070065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9070128Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9070407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9070471Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9070682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9070783Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9071012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9071088Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9071315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9071385Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9071620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9071733Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9071736Z 
2025-09-07T08:14:18.9071831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9072013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9072071Z     return mod(**inputs)
2025-09-07T08:14:18.9072332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9072394Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9072657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9072718Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9072933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9072998Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9073225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9073294Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9073519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9073592Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9073817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9073965Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9073969Z 
2025-09-07T08:14:18.9074056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9074237Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9074295Z     return mod(**inputs)
2025-09-07T08:14:18.9074525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9074588Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9074816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9074876Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9075092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9075156Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9075385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9075451Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9075680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9075782Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9076009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9076146Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9076150Z 
2025-09-07T08:14:18.9076272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9076456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9076510Z     return mod(**inputs)
2025-09-07T08:14:18.9076740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9076804Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9077030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9077095Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9077307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9077370Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9077598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9077700Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9077934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9078005Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9078267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9078336Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9078340Z 
2025-09-07T08:14:18.9078425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9078612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9078667Z     return mod(**inputs)
2025-09-07T08:14:18.9078898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9078959Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9079187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9079252Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9079466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9079536Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9079759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9079827Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9080060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9080129Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9080361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9080464Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9080468Z 
2025-09-07T08:14:18.9080559Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9080743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9080799Z     return mod(**inputs)
2025-09-07T08:14:18.9081032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9081094Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9081327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9081421Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9081633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9081703Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9081965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9082036Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9082262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9082335Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9082559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9082651Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9082656Z 
2025-09-07T08:14:18.9082751Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9082942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9083003Z     return mod(**inputs)
2025-09-07T08:14:18.9083534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9083600Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9083866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9083926Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9084141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9084207Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9084433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9084511Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9084735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9084808Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9085037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9085133Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9085136Z 
2025-09-07T08:14:18.9085225Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9085407Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9085466Z     return mod(**inputs)
2025-09-07T08:14:18.9085696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9085762Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9085993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9086054Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9086274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9086341Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9086571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9086641Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9086869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9086944Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9087169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9087296Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9087300Z 
2025-09-07T08:14:18.9087386Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9087568Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9087672Z     return mod(**inputs)
2025-09-07T08:14:18.9087900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9087965Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9088195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9088259Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9088470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9088540Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9088772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9088838Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9089098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9089169Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9089429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9089498Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9089502Z 
2025-09-07T08:14:18.9089572Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9089671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9089859Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9089922Z     return mod(**inputs)
2025-09-07T08:14:18.9090156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9090222Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9090459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9090522Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9090742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9090815Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9091050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9091135Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9091363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9091454Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9091680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9091753Z     return self.weight * hidden_states
2025-09-07T08:14:18.9091758Z 
2025-09-07T08:14:18.9091848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9092033Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9092093Z     return mod(**inputs)
2025-09-07T08:14:18.9092325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9092388Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9092615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9092711Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9092930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9092995Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9093230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9093342Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9093573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9093691Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9093920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9093998Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9094003Z 
2025-09-07T08:14:18.9094091Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9094279Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9094335Z     return mod(**inputs)
2025-09-07T08:14:18.9094567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9094669Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9094903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9095001Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9095217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9095284Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9095518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9095598Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9095833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9095936Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9096166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9096241Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9096244Z 
2025-09-07T08:14:18.9096333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9096521Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9096580Z     return mod(**inputs)
2025-09-07T08:14:18.9096813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9096878Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9097105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9097229Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9097443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9097520Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9097746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9097822Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9098053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9098150Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9098378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9098484Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9098487Z 
2025-09-07T08:14:18.9098565Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9098654Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9098838Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9098933Z     return mod(**inputs)
2025-09-07T08:14:18.9099164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9099232Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9099462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9099522Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9099736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9099803Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9100033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9100102Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9100362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9100462Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9100719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9100787Z     return self.weight * hidden_states
2025-09-07T08:14:18.9100790Z 
2025-09-07T08:14:18.9100877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9101060Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9101118Z     return mod(**inputs)
2025-09-07T08:14:18.9101346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9101411Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9101641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9101708Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9101919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9101984Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9102218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9102291Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9102536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9102614Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9102840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9102916Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9102920Z 
2025-09-07T08:14:18.9103010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9103365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9103421Z     return mod(**inputs)
2025-09-07T08:14:18.9103655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9103716Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9103942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9104074Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9104284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9104352Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9104578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9104697Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9104925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9104996Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9105229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9105295Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9105299Z 
2025-09-07T08:14:18.9105394Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9105579Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9105636Z     return mod(**inputs)
2025-09-07T08:14:18.9105873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9105935Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9106215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9106279Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9106536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9106611Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9106838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9106913Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9107136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9107206Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9107439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9107560Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9107563Z 
2025-09-07T08:14:18.9107653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9107836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9107896Z     return mod(**inputs)
2025-09-07T08:14:18.9108126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9108187Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9108420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9108481Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9108692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9108759Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9108986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9109059Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9109286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9109358Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9109586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9109746Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9109751Z 
2025-09-07T08:14:18.9109841Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9110022Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9110082Z     return mod(**inputs)
2025-09-07T08:14:18.9110350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9110418Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9110651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9110714Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9110930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9110998Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9111231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9111299Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9111525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9111634Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9111861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9112008Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9112011Z 
2025-09-07T08:14:18.9112101Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9112285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9112341Z     return mod(**inputs)
2025-09-07T08:14:18.9112571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9112640Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9112868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9112933Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9113146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9113214Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9113444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9113513Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9113742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9113811Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9114037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9114184Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9114187Z 
2025-09-07T08:14:18.9114274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9114463Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9114521Z     return mod(**inputs)
2025-09-07T08:14:18.9114751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9114812Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9115039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9115101Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9115351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9115422Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9115653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9115761Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9115996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9116067Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9116302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9116444Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9116448Z 
2025-09-07T08:14:18.9116541Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9116731Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9116789Z     return mod(**inputs)
2025-09-07T08:14:18.9117026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9117088Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9117355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9117419Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9117677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9117748Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9117985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9118058Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9118285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9118355Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9118581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9118653Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9118656Z 
2025-09-07T08:14:18.9118749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9118932Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9118990Z     return mod(**inputs)
2025-09-07T08:14:18.9119218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9119280Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9119512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9119573Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9119787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9119851Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9120080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9120153Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9120379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9120452Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9120676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9120777Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9120818Z 
2025-09-07T08:14:18.9120910Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9121093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9121153Z     return mod(**inputs)
2025-09-07T08:14:18.9121382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9121479Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9121709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9121772Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9121984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9122050Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9122279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9122351Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9122579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9122650Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9122909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9123005Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9123008Z 
2025-09-07T08:14:18.9123124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9123311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9123366Z     return mod(**inputs)
2025-09-07T08:14:18.9123597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9123663Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9123890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9123952Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9124165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9124231Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9124458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9124528Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9124754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9124822Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9125047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9125148Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9125151Z 
2025-09-07T08:14:18.9125241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9125429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9125489Z     return mod(**inputs)
2025-09-07T08:14:18.9125727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9125792Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9126021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9126090Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9126302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9126409Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9126638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9126708Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9126964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9127068Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9127305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9127398Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9127401Z 
2025-09-07T08:14:18.9127492Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9127684Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9127742Z     return mod(**inputs)
2025-09-07T08:14:18.9127983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9128048Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9128285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9128382Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9128597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9128717Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9128951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9129027Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9129253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9129327Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9129558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9129628Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9129632Z 
2025-09-07T08:14:18.9129730Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9129914Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9129972Z     return mod(**inputs)
2025-09-07T08:14:18.9130211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9130275Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9130509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9130570Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9130790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9130858Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9131087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9131165Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9131392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.9131523Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9131527Z 
2025-09-07T08:14:18.9131600Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9131692Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9131889Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9131980Z     return mod(**inputs)
2025-09-07T08:14:18.9132236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9132300Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9132532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9132636Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9132852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9132928Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9133158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9133247Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9133477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9133566Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9133797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9133865Z     return self.weight * hidden_states
2025-09-07T08:14:18.9133869Z 
2025-09-07T08:14:18.9134002Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9134193Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9134251Z     return mod(**inputs)
2025-09-07T08:14:18.9134522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9134585Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9134819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9134885Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9135103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9135171Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9135398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9135488Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9135716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9135830Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9136057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9136129Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9136132Z 
2025-09-07T08:14:18.9136227Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9136414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9136476Z     return mod(**inputs)
2025-09-07T08:14:18.9136706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9136773Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9137013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9137075Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9137344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9137412Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9137643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9137781Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9138010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9138120Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9138348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9138462Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9138465Z 
2025-09-07T08:14:18.9138552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9138739Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9138803Z     return mod(**inputs)
2025-09-07T08:14:18.9139034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9139102Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9139333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9139395Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9139612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9139712Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9139949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9140027Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9140293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9140394Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9140624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9140704Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9140707Z 
2025-09-07T08:14:18.9140781Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9140875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9141057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9141118Z     return mod(**inputs)
2025-09-07T08:14:18.9141351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T08:14:18.9141414Z     encoder_outputs = self.encoder(
2025-09-07T08:14:18.9141652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward
2025-09-07T08:14:18.9141748Z     hidden_states = self.final_layer_norm(hidden_states)
2025-09-07T08:14:18.9141982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9142052Z     return self.weight * hidden_states
2025-09-07T08:14:18.9142055Z 
2025-09-07T08:14:18.9142150Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9142343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9142401Z     return mod(**inputs)
2025-09-07T08:14:18.9142646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9142712Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9142945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9143014Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9143226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9143300Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9143573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9143647Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9143883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9143996Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9144229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9144299Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9144303Z 
2025-09-07T08:14:18.9144401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9144586Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9144645Z     return mod(**inputs)
2025-09-07T08:14:18.9144883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9144949Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9145184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9145247Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9145499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9145574Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9145833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9145910Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9146137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9146211Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9146443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9146562Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9146566Z 
2025-09-07T08:14:18.9146662Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9146848Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9146913Z     return mod(**inputs)
2025-09-07T08:14:18.9147142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9147207Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9147447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9147509Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9147728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9147795Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9148024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9148097Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9148326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9148404Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9148631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9148740Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9148749Z 
2025-09-07T08:14:18.9148840Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9149024Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9149121Z     return mod(**inputs)
2025-09-07T08:14:18.9149351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9149421Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9149692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9149756Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9149979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9150046Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9150282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9150353Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9150582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9150657Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9150884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9151050Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9151054Z 
2025-09-07T08:14:18.9151142Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9151358Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9151415Z     return mod(**inputs)
2025-09-07T08:14:18.9151646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9151716Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9151944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9152013Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9152224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9152291Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9152525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9152596Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9152828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9152900Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9153127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9153280Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9153285Z 
2025-09-07T08:14:18.9153375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9153562Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9153619Z     return mod(**inputs)
2025-09-07T08:14:18.9153856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9153919Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9154151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9154215Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9154428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9154500Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9154727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9154831Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9155062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9155135Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9155408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9155477Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9155480Z 
2025-09-07T08:14:18.9155575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9155757Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9155816Z     return mod(**inputs)
2025-09-07T08:14:18.9156048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9156112Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9156347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9156407Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9156653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9156728Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9156988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9157063Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9157289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9157362Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9157593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9157694Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9157697Z 
2025-09-07T08:14:18.9157793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9157975Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9158038Z     return mod(**inputs)
2025-09-07T08:14:18.9158267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9158331Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9158566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9158632Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9158849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9158917Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9159144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9159220Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9159447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9159525Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9159750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9159842Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9159851Z 
2025-09-07T08:14:18.9159938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9160121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9160220Z     return mod(**inputs)
2025-09-07T08:14:18.9160450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9160516Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9160745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9160843Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9161056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9161122Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9161353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9161424Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9161652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9161730Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9161957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9162053Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9162058Z 
2025-09-07T08:14:18.9162177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9162368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9162425Z     return mod(**inputs)
2025-09-07T08:14:18.9162686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9162760Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9162992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9163061Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9163271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9163339Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9163572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9163642Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9163877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9163951Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9164177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9164272Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9164275Z 
2025-09-07T08:14:18.9164363Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9164553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9164608Z     return mod(**inputs)
2025-09-07T08:14:18.9164840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9164905Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9165139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9165207Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9165419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9165490Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9165719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9165828Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9166062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9166131Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9166358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9166459Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9166462Z 
2025-09-07T08:14:18.9166534Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9166623Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9166803Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9166862Z     return mod(**inputs)
2025-09-07T08:14:18.9167091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9167157Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9167386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9167445Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9167659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9167757Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9167990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9168104Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9168335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9168424Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9168651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9168725Z     return self.weight * hidden_states
2025-09-07T08:14:18.9168728Z 
2025-09-07T08:14:18.9168817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9169003Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9169060Z     return mod(**inputs)
2025-09-07T08:14:18.9169288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9169354Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9169583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9169647Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9169857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9169925Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9170154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9170231Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9170458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9170568Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9170791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9170864Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9170867Z 
2025-09-07T08:14:18.9170954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9171137Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9171192Z     return mod(**inputs)
2025-09-07T08:14:18.9171459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9171521Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9171749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9171863Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9172073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9172140Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9172366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9172440Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9172685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9172787Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9173019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9173091Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9173094Z 
2025-09-07T08:14:18.9173185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9173407Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9173466Z     return mod(**inputs)
2025-09-07T08:14:18.9173732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9173797Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9174033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9174095Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9174308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9174382Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9174611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9174693Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9174921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9175023Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9175256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9175324Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9175327Z 
2025-09-07T08:14:18.9175422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9175605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9175667Z     return mod(**inputs)
2025-09-07T08:14:18.9175896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9175959Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9176195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9176255Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9176470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9176536Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9176764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9176840Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9177104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9177261Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9177488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9177601Z     return self.weight * hidden_states
2025-09-07T08:14:18.9177609Z 
2025-09-07T08:14:18.9177698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9177883Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9177944Z     return mod(**inputs)
2025-09-07T08:14:18.9178176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9178244Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9178483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9178548Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9178765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9178833Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9179109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9179183Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9179443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9179524Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9179749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9179822Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9179828Z 
2025-09-07T08:14:18.9179917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9180099Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9180162Z     return mod(**inputs)
2025-09-07T08:14:18.9180393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9180465Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9180696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9180761Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9180971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9181037Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9181268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9181340Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9181572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9181643Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9181872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9181947Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9181951Z 
2025-09-07T08:14:18.9182040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9182228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9182286Z     return mod(**inputs)
2025-09-07T08:14:18.9182514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9182620Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9182852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9182920Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9183130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9183237Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9183463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9183536Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9183767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9183839Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9184072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9184190Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9184193Z 
2025-09-07T08:14:18.9184279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9184468Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9184558Z     return mod(**inputs)
2025-09-07T08:14:18.9184799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9184897Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9185137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9185198Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9185415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9185492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9185720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9185797Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9186023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9186096Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9186332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9186445Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9186448Z 
2025-09-07T08:14:18.9186543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9186728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9186787Z     return mod(**inputs)
2025-09-07T08:14:18.9187025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9187089Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9187322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9187386Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9187605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9187674Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9187901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9187975Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9188203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9188314Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9188544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9188653Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9188688Z 
2025-09-07T08:14:18.9188788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9188972Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9189032Z     return mod(**inputs)
2025-09-07T08:14:18.9189263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9189332Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9189563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9189625Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9189842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9189911Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9190143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9190245Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9190477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9190584Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9190813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9190965Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9190969Z 
2025-09-07T08:14:18.9191059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9191244Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9191308Z     return mod(**inputs)
2025-09-07T08:14:18.9191539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9191610Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9191838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9191904Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9192119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9192185Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9192417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9192490Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9192719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9192789Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9193017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9193089Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9193092Z 
2025-09-07T08:14:18.9193181Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9193371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9193428Z     return mod(**inputs)
2025-09-07T08:14:18.9193658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9193727Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9194013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9194083Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9194293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9194408Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9194636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9194708Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9194944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9195015Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9195245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9195344Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9195348Z 
2025-09-07T08:14:18.9195438Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9195628Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9195687Z     return mod(**inputs)
2025-09-07T08:14:18.9195958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9196024Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9196291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9196355Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9196574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9196651Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9196880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9196956Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9197182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9197255Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9197487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9197582Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9197585Z 
2025-09-07T08:14:18.9197680Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9197861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9197918Z     return mod(**inputs)
2025-09-07T08:14:18.9198153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9198217Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9198450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9198511Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9198734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9198802Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9199031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9199109Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9199334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9199408Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9199671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9199766Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9199769Z 
2025-09-07T08:14:18.9199864Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9200083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9200147Z     return mod(**inputs)
2025-09-07T08:14:18.9200380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9200443Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9200679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9200741Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9200963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9201032Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9201264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9201333Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9201595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9201674Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9201934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9202034Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9202037Z 
2025-09-07T08:14:18.9202125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9202315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9202377Z     return mod(**inputs)
2025-09-07T08:14:18.9202609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9202679Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9202908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9202980Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9203358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9203433Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9203688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9203759Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9203996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9204071Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9204300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9204374Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9204381Z 
2025-09-07T08:14:18.9204455Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9204556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9204755Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9204816Z     return mod(**inputs)
2025-09-07T08:14:18.9205057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9205122Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9205361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9205482Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9205699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9205778Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9206056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9206135Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9206363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9206470Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9206698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9206768Z     return self.weight * hidden_states
2025-09-07T08:14:18.9206774Z 
2025-09-07T08:14:18.9206873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9207063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9207127Z     return mod(**inputs)
2025-09-07T08:14:18.9207401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9207469Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9207748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9207814Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9208036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9208104Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9208337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9208411Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9208641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9208721Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9208955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9209032Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9209035Z 
2025-09-07T08:14:18.9209127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9209314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9209377Z     return mod(**inputs)
2025-09-07T08:14:18.9209611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9209683Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9209913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9209976Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9210197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9210268Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9210501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9210574Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9210810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9210884Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9211111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9211220Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9211223Z 
2025-09-07T08:14:18.9211315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9211503Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9211594Z     return mod(**inputs)
2025-09-07T08:14:18.9211826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9211896Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9212126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9212193Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9212406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9212476Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9212710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9212782Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9213050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9213125Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9213360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9213527Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9213530Z 
2025-09-07T08:14:18.9213621Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9213813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9213872Z     return mod(**inputs)
2025-09-07T08:14:18.9214110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9214174Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9214404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9214473Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9214689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9214764Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9214995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9215065Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9215305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9215380Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9215613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9215725Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9215729Z 
2025-09-07T08:14:18.9215828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9216013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9216072Z     return mod(**inputs)
2025-09-07T08:14:18.9216310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9216373Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9216613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9216707Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9216920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9216994Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9217270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9217382Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9217609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9217689Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9217915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9218024Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9218028Z 
2025-09-07T08:14:18.9218118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9218304Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9218363Z     return mod(**inputs)
2025-09-07T08:14:18.9218592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9218690Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9218931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9218995Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9219245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9219312Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9219542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9219617Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9219843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9219918Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9220144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9220294Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9220298Z 
2025-09-07T08:14:18.9220386Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9220568Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9220627Z     return mod(**inputs)
2025-09-07T08:14:18.9220857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9220925Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9221155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9221217Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9221434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9221504Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9221737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9221810Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9222041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9222114Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9222340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9222455Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9222458Z 
2025-09-07T08:14:18.9222550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9222742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9222835Z     return mod(**inputs)
2025-09-07T08:14:18.9223066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9223136Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9223370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9223439Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9223652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9223719Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9223954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9224025Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9224258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9224363Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9224599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9224756Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9224759Z 
2025-09-07T08:14:18.9224852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9225041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9225097Z     return mod(**inputs)
2025-09-07T08:14:18.9225335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9225398Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9225627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9225698Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9225911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9225987Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9226220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9226293Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9226526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9226598Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9226835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9226927Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9226930Z 
2025-09-07T08:14:18.9227025Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9227212Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9227270Z     return mod(**inputs)
2025-09-07T08:14:18.9227508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9227569Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9227807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9227870Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9228317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9228392Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9228620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9228699Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9228966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9229047Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9229275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9229366Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9229370Z 
2025-09-07T08:14:18.9229466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9229650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9229714Z     return mod(**inputs)
2025-09-07T08:14:18.9229943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9230007Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9230278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9230341Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9230595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9230665Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9230894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9230974Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9231200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9231279Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9231506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9231609Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9231612Z 
2025-09-07T08:14:18.9231701Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9231884Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9231949Z     return mod(**inputs)
2025-09-07T08:14:18.9232178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9232249Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9232489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9232556Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9232775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9232846Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9233083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9233153Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9233383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9237341Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9237639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9237721Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9237804Z 
2025-09-07T08:14:18.9237886Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9237988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9238202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9238263Z     return mod(**inputs)
2025-09-07T08:14:18.9238564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9238632Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9238879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9238949Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9239185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9239264Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9239513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9239602Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9239843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9239989Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9240243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9240316Z     return self.weight * hidden_states
2025-09-07T08:14:18.9240350Z 
2025-09-07T08:14:18.9240455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9240659Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9240723Z     return mod(**inputs)
2025-09-07T08:14:18.9240969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9241038Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9241272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9241338Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9241559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9241636Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9241867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9241954Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9242224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9242348Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9242590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9242664Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9242667Z 
2025-09-07T08:14:18.9242767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9242969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9243031Z     return mod(**inputs)
2025-09-07T08:14:18.9243268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9243335Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9243570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9243634Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9243855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9243964Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9244199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9244279Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9244545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9244656Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9244883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9244959Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9244962Z 
2025-09-07T08:14:18.9245056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9245246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9245314Z     return mod(**inputs)
2025-09-07T08:14:18.9245545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9245610Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9245870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9245937Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9246155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9246255Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9246491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9246569Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9246799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9246903Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9247128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9247203Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9247209Z 
2025-09-07T08:14:18.9247284Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9247382Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9247575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9247632Z     return mod(**inputs)
2025-09-07T08:14:18.9247867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9247930Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9248166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9248229Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9248443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9248516Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9248745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9248821Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9249048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9249146Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9249373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9249483Z     return self.weight * hidden_states
2025-09-07T08:14:18.9249486Z 
2025-09-07T08:14:18.9249580Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9249765Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9249828Z     return mod(**inputs)
2025-09-07T08:14:18.9250064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9250160Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9250400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9250464Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9250680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9250747Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9250975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9251052Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9251294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9251378Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9251642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9251717Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9251721Z 
2025-09-07T08:14:18.9251842Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9252032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9252095Z     return mod(**inputs)
2025-09-07T08:14:18.9252326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9252398Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9252627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9252693Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9252912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9252984Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9253223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9253297Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9253529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9253602Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9253828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9253904Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9253908Z 
2025-09-07T08:14:18.9254002Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9254190Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9254250Z     return mod(**inputs)
2025-09-07T08:14:18.9254480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9254550Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9254781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9254849Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9255062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9255166Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9255398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9255470Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9255707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9255810Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9256044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9256169Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9256173Z 
2025-09-07T08:14:18.9256264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9256454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9256514Z     return mod(**inputs)
2025-09-07T08:14:18.9256744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9256807Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9257046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9257150Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9257438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9257546Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9257782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9257856Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9258090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9258166Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9258392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9258510Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9258515Z 
2025-09-07T08:14:18.9258618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9258808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9258864Z     return mod(**inputs)
2025-09-07T08:14:18.9259105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9259170Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9259404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9259468Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9259683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9259755Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9259984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9260059Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9260283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9260358Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9260583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9260694Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9260697Z 
2025-09-07T08:14:18.9260789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9261010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9261068Z     return mod(**inputs)
2025-09-07T08:14:18.9261299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9261410Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9261640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9261700Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9261916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9261982Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9262208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9262281Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9262511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9262581Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9262807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9262990Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9262993Z 
2025-09-07T08:14:18.9263084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9263299Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9263359Z     return mod(**inputs)
2025-09-07T08:14:18.9263589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9263653Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9263883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9263943Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9264157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9264226Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9264454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9264523Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9264749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9264821Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9265044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9265116Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9265120Z 
2025-09-07T08:14:18.9265207Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9265392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9265446Z     return mod(**inputs)
2025-09-07T08:14:18.9265677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9265744Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9265983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9266050Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9266262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9266326Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9266603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9266675Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9266909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9267015Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9267250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9267355Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9267358Z 
2025-09-07T08:14:18.9267448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9267642Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9267699Z     return mod(**inputs)
2025-09-07T08:14:18.9267937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9268003Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9268232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9268297Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9268546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9268624Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9268884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9268955Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9269185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9269257Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9269487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9269579Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9269582Z 
2025-09-07T08:14:18.9269674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9269861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9269918Z     return mod(**inputs)
2025-09-07T08:14:18.9270153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9270214Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9270448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9270509Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9270720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9270789Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9271016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9271089Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9271318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9271387Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9271616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9271703Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9271707Z 
2025-09-07T08:14:18.9271799Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9271979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9272068Z     return mod(**inputs)
2025-09-07T08:14:18.9272299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9272360Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9272597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9272694Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9272910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9272976Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9273204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9273275Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9273498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9273575Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9273802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9273899Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9273936Z 
2025-09-07T08:14:18.9274027Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9274208Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9274302Z     return mod(**inputs)
2025-09-07T08:14:18.9274535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9274601Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9274831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9274896Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9275110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9275176Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9275408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9275483Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9275712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9275790Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9276015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9276089Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9276092Z 
2025-09-07T08:14:18.9276185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9276371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9276428Z     return mod(**inputs)
2025-09-07T08:14:18.9276658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9276731Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9276958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9277028Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9277239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9277306Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9277533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9277637Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9277872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.9277997Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9278033Z 
2025-09-07T08:14:18.9278110Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9278201Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9278387Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9278450Z     return mod(**inputs)
2025-09-07T08:14:18.9278682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9278749Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9278981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9279046Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9279265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9279332Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9279600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9279676Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9279934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9280037Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9280265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9280340Z     return self.weight * hidden_states
2025-09-07T08:14:18.9280345Z 
2025-09-07T08:14:18.9280435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9280623Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9280680Z     return mod(**inputs)
2025-09-07T08:14:18.9280912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9280984Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9281214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9281283Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9281496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9281564Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9281796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9281868Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9282102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9282175Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9282402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9282480Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9282483Z 
2025-09-07T08:14:18.9282571Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9282761Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9282817Z     return mod(**inputs)
2025-09-07T08:14:18.9283055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9283169Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9283399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9283466Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9283678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9283785Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9284011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9284083Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9284316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9284390Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9284622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9284693Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9284697Z 
2025-09-07T08:14:18.9284785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9284969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9285027Z     return mod(**inputs)
2025-09-07T08:14:18.9285293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9285357Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9285618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9285679Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9285890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9285963Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9286191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9286265Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9286495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9286568Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9286802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9286923Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9286927Z 
2025-09-07T08:14:18.9287019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9287200Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9287256Z     return mod(**inputs)
2025-09-07T08:14:18.9287490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9287553Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9287784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9287847Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9288068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9288135Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9288368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9288439Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9288664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9288772Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9288997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9289111Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9289116Z 
2025-09-07T08:14:18.9289213Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9289435Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9289497Z     return mod(**inputs)
2025-09-07T08:14:18.9289733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9289797Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9290033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9290096Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9290317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9290382Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9290608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9290713Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9290947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9291021Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9291276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9291394Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9291397Z 
2025-09-07T08:14:18.9291486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9291680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9291738Z     return mod(**inputs)
2025-09-07T08:14:18.9291969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9292037Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9292272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9292338Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9292552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9292620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9292852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9292923Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9293155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9293227Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9293459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9293610Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9293613Z 
2025-09-07T08:14:18.9293704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9293894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9293953Z     return mod(**inputs)
2025-09-07T08:14:18.9294186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9294250Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9294519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9294586Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9294799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9294870Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9295131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9295204Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9295437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9295510Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9295743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9295814Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9295818Z 
2025-09-07T08:14:18.9295910Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9296092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9296150Z     return mod(**inputs)
2025-09-07T08:14:18.9296414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9296479Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9296751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9296814Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9297026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9297099Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9297392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9297468Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9297693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9297773Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9298002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9298100Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9298103Z 
2025-09-07T08:14:18.9298204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9298389Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9298451Z     return mod(**inputs)
2025-09-07T08:14:18.9298681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9298747Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9298984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9299045Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9299260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9299329Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9299560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9299637Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9299862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9299941Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9300211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9300308Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9300311Z 
2025-09-07T08:14:18.9300398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9300585Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9300683Z     return mod(**inputs)
2025-09-07T08:14:18.9300915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9300979Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9301209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9301270Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9301489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9301556Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9301788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9301859Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9302122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9302201Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9302612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9302706Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9302710Z 
2025-09-07T08:14:18.9302799Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9302986Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9303273Z     return mod(**inputs)
2025-09-07T08:14:18.9303535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9303602Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9303837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9303905Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9304117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9304182Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9304413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9304481Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9304714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9304785Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9305017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9305117Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9305124Z 
2025-09-07T08:14:18.9305215Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9305399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9305456Z     return mod(**inputs)
2025-09-07T08:14:18.9305693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9305756Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9305987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9306125Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9306338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9306406Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9306637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9306755Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9306989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9307062Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9307292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9307361Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9307364Z 
2025-09-07T08:14:18.9307439Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9307528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9307710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9307773Z     return mod(**inputs)
2025-09-07T08:14:18.9308047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9308119Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9308348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9308455Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9308672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9308741Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9308975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9309058Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9309285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9309376Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9309606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9309679Z     return self.weight * hidden_states
2025-09-07T08:14:18.9309682Z 
2025-09-07T08:14:18.9309775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9309960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9310016Z     return mod(**inputs)
2025-09-07T08:14:18.9310245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9310316Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9310545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9310612Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9310823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9310892Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9311124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9311205Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9311437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9311546Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9311786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9311899Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9311902Z 
2025-09-07T08:14:18.9311993Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9312181Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9312271Z     return mod(**inputs)
2025-09-07T08:14:18.9312506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9312572Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9312803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9312870Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9313080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9313152Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9313381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9313459Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9313721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9313825Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9314085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9314160Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9314163Z 
2025-09-07T08:14:18.9314254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9314438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9314495Z     return mod(**inputs)
2025-09-07T08:14:18.9314730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9314790Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9315024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9315088Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9315300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9315374Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9315602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9315682Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9315905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9316006Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9316237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9316304Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9316309Z 
2025-09-07T08:14:18.9316386Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9316475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9316664Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9316722Z     return mod(**inputs)
2025-09-07T08:14:18.9316952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9317018Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9317247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9317348Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9317561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9317628Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9317871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9317976Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9318207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9318305Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9318531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9318605Z     return self.weight * hidden_states
2025-09-07T08:14:18.9318609Z 
2025-09-07T08:14:18.9318699Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9318887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9318944Z     return mod(**inputs)
2025-09-07T08:14:18.9319175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9319311Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9319546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9319641Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9319866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9319940Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9320167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9320241Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9320479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9320555Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9320799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9320868Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9320872Z 
2025-09-07T08:14:18.9320965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9321148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9321203Z     return mod(**inputs)
2025-09-07T08:14:18.9321435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9321498Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9321731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9321793Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9322003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9322076Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9322304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9322377Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9322600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9322669Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9322898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9323000Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9323003Z 
2025-09-07T08:14:18.9323096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9323278Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9323368Z     return mod(**inputs)
2025-09-07T08:14:18.9323601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9323664Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9323900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9323959Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9324173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9324240Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9324468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9324542Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9324769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9324893Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9325122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9325269Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9325277Z 
2025-09-07T08:14:18.9325364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9325544Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9325604Z     return mod(**inputs)
2025-09-07T08:14:18.9325834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9325902Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9326129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9326192Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9326408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9326475Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9326707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9326774Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9327000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9327076Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9327304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9327419Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9327422Z 
2025-09-07T08:14:18.9327509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9327696Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9327752Z     return mod(**inputs)
2025-09-07T08:14:18.9327981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9328047Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9328276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9328342Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9328593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9328661Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9328893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9328998Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9329229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9329298Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9329524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9329640Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9329643Z 
2025-09-07T08:14:18.9329731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9329919Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9329973Z     return mod(**inputs)
2025-09-07T08:14:18.9330203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9330263Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9330529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9330597Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9330840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9330911Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9331139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9331208Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9331443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9331512Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9331742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9331891Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9331895Z 
2025-09-07T08:14:18.9331988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9332173Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9332230Z     return mod(**inputs)
2025-09-07T08:14:18.9332465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9332526Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9332762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9332824Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9333035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9333102Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9333330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9333406Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9333644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9333715Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9333942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9334011Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9334049Z 
2025-09-07T08:14:18.9334145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9334327Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9334386Z     return mod(**inputs)
2025-09-07T08:14:18.9334617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9334711Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9334951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9335014Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9335233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9335299Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9335523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9335600Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9335829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9335904Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9336163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9336263Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9336271Z 
2025-09-07T08:14:18.9336389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9336573Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9336636Z     return mod(**inputs)
2025-09-07T08:14:18.9336867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9336932Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9337221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9337284Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9337498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9337565Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9337799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9337868Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9338094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9338170Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9338401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9338502Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9338504Z 
2025-09-07T08:14:18.9338598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9338790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9338855Z     return mod(**inputs)
2025-09-07T08:14:18.9339092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9339161Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9339394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9339463Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9339679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9339790Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9340027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9340099Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9340335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9340440Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9340669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9340770Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9340774Z 
2025-09-07T08:14:18.9340863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9341052Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9341111Z     return mod(**inputs)
2025-09-07T08:14:18.9341346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9341411Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9341641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9341746Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9341963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9342063Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9342295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9342366Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9342600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9342672Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9342903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9342996Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9343001Z 
2025-09-07T08:14:18.9343093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9343287Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9343343Z     return mod(**inputs)
2025-09-07T08:14:18.9343582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9343651Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9343880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9343941Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9344151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9344219Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9344445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9344517Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9344741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9344809Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9345037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9345104Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9345107Z 
2025-09-07T08:14:18.9345181Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9345303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9345484Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9345542Z     return mod(**inputs)
2025-09-07T08:14:18.9345770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9345887Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9346116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9346179Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9346389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9346455Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9346685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9346755Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9346982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9347076Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9347340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9347410Z     return self.weight * hidden_states
2025-09-07T08:14:18.9347414Z 
2025-09-07T08:14:18.9347499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9347723Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9347779Z     return mod(**inputs)
2025-09-07T08:14:18.9348009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9348077Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9348307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9348369Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9348581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9348650Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9348877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9348950Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9349178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9349251Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9349481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9349549Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9349552Z 
2025-09-07T08:14:18.9349640Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9349825Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9349883Z     return mod(**inputs)
2025-09-07T08:14:18.9350114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9350176Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9350428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9350492Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9350704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9350776Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9351040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9351112Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9351336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9351445Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9351674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9351743Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9351747Z 
2025-09-07T08:14:18.9351839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9352020Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9352074Z     return mod(**inputs)
2025-09-07T08:14:18.9352304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9352367Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9352598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9352657Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9352903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9352971Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9353242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9353314Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9353540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9353612Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9353839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9353956Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9353960Z 
2025-09-07T08:14:18.9354051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9354235Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9354292Z     return mod(**inputs)
2025-09-07T08:14:18.9354521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9354582Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9354813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9354873Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9355085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9355150Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9355379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9355447Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9355677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9355749Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9355973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9356083Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9356087Z 
2025-09-07T08:14:18.9356173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9356353Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9356446Z     return mod(**inputs)
2025-09-07T08:14:18.9356675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9356738Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9357005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9357066Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9357281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9357346Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9357573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9357641Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9357869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9357940Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9358164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9358308Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9358311Z 
2025-09-07T08:14:18.9358399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9358621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9358682Z     return mod(**inputs)
2025-09-07T08:14:18.9358922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9358991Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9359222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9359290Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9359506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9359580Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9359817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9359889Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9360119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9360191Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9360418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9360566Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9360571Z 
2025-09-07T08:14:18.9360679Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9360875Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9360932Z     return mod(**inputs)
2025-09-07T08:14:18.9361168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9361231Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9361461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9361525Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9361739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9361807Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9362031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9362137Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9362366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9362436Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9362698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9362769Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9362773Z 
2025-09-07T08:14:18.9362868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9363058Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9363118Z     return mod(**inputs)
2025-09-07T08:14:18.9363361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9363430Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9363668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9363731Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9363981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9364058Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9364323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9364397Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9364625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9364700Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9364924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9365027Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9365030Z 
2025-09-07T08:14:18.9365129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9365323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9365383Z     return mod(**inputs)
2025-09-07T08:14:18.9365615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9365679Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9365908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9365971Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9366187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9366255Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9366483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9366552Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9366777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9366854Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9367085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9367179Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9367182Z 
2025-09-07T08:14:18.9367272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9367465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9367578Z     return mod(**inputs)
2025-09-07T08:14:18.9367810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9367875Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9368110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9368209Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9368430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9368499Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9368734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9368804Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9369036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9369111Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9369339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9369443Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9369448Z 
2025-09-07T08:14:18.9369578Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9369769Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9369826Z     return mod(**inputs)
2025-09-07T08:14:18.9370090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9370163Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9370396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9370467Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9370681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9370752Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9370988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9371062Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9371295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9371367Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9371604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9371701Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9371704Z 
2025-09-07T08:14:18.9371797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9371993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9372053Z     return mod(**inputs)
2025-09-07T08:14:18.9372293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9372360Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9372610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9372680Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9372895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9372971Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9373197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9373307Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9373536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9373610Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9373844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9373952Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9373956Z 
2025-09-07T08:14:18.9374052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9374239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9374295Z     return mod(**inputs)
2025-09-07T08:14:18.9374531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9374595Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9374830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9374892Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9375106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9375219Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9375451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9375529Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9375789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T08:14:18.9375921Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9375924Z 
2025-09-07T08:14:18.9375998Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9376094Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9376287Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9376346Z     return mod(**inputs)
2025-09-07T08:14:18.9376583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9376650Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9376880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9376949Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9377203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9377282Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9377513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9377598Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9377832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9377922Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9378154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9378225Z     return self.weight * hidden_states
2025-09-07T08:14:18.9378228Z 
2025-09-07T08:14:18.9378321Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9378507Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9378565Z     return mod(**inputs)
2025-09-07T08:14:18.9378802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9378866Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9379146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9379211Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9379424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9379533Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9379762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9379849Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9380075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9380186Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9380419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9380495Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9380498Z 
2025-09-07T08:14:18.9380592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9380776Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9380841Z     return mod(**inputs)
2025-09-07T08:14:18.9381105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9381171Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9381443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9381507Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9381723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9381792Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9382022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9382104Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9382331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9382447Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9382673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9382753Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9382757Z 
2025-09-07T08:14:18.9382847Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9383032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9383095Z     return mod(**inputs)
2025-09-07T08:14:18.9383325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9383396Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9383626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9383689Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9383910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9383979Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9384209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9384282Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9384510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9384652Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9384879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9384954Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9384957Z 
2025-09-07T08:14:18.9385027Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9385160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9385343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9385401Z     return mod(**inputs)
2025-09-07T08:14:18.9385644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9385709Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9385946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9386011Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9386224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9386297Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9386527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9386641Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9386876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9387021Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9387256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9387325Z     return self.weight * hidden_states
2025-09-07T08:14:18.9387328Z 
2025-09-07T08:14:18.9387426Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9387609Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9387674Z     return mod(**inputs)
2025-09-07T08:14:18.9387903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9387970Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9388208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9388270Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9388488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9388555Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9388782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9388861Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9389090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9389170Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9389399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9389471Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9389480Z 
2025-09-07T08:14:18.9389568Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9389753Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9389819Z     return mod(**inputs)
2025-09-07T08:14:18.9390050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9390118Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9390386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9390449Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9390669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9390737Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9391008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9391077Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9391304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9391375Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9391601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9391673Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9391676Z 
2025-09-07T08:14:18.9391763Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9391945Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9392000Z     return mod(**inputs)
2025-09-07T08:14:18.9392259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9392324Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9392586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9392649Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9392859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9392925Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9393151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9393222Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9393453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9393520Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9393748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9393870Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9393874Z 
2025-09-07T08:14:18.9393964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9394147Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9394202Z     return mod(**inputs)
2025-09-07T08:14:18.9394431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9394493Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9394720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9394782Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9394994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9395061Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9395288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9395357Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9395585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9395654Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9395917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9396030Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9396034Z 
2025-09-07T08:14:18.9396126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9396312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9396398Z     return mod(**inputs)
2025-09-07T08:14:18.9396631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9396701Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9396939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9397003Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9397220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9397294Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9397524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9397601Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9397862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9397938Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9398212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9398324Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9398327Z 
2025-09-07T08:14:18.9398425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9398607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9398674Z     return mod(**inputs)
2025-09-07T08:14:18.9398905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9398969Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9399209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9399271Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9399485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9399554Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9399784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9399863Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9400091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9400171Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9400404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9400551Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9400564Z 
2025-09-07T08:14:18.9400656Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9400841Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9400907Z     return mod(**inputs)
2025-09-07T08:14:18.9401136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9401207Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9401437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9401535Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9401755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9401822Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9402057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9402165Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9402396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9402477Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9402704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9402778Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9402783Z 
2025-09-07T08:14:18.9402875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9403205Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9403262Z     return mod(**inputs)
2025-09-07T08:14:18.9403494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9403620Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9403864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9403976Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9404203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9404279Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9404519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9404596Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9404824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9404895Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9405121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9405226Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9405229Z 
2025-09-07T08:14:18.9405327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9405518Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9405574Z     return mod(**inputs)
2025-09-07T08:14:18.9405814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9405883Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9406112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9406179Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9406396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9406470Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9406701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9406775Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9407008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9407077Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9407313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9407454Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9407458Z 
2025-09-07T08:14:18.9407557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9407741Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9407869Z     return mod(**inputs)
2025-09-07T08:14:18.9408109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9408169Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9408405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9408465Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9408683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9408767Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9408999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9409074Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9409300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9409422Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9409659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9409782Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9409786Z 
2025-09-07T08:14:18.9409885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9410069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9410135Z     return mod(**inputs)
2025-09-07T08:14:18.9410371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9410432Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9410667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9410729Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9410951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9411016Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9411247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9411318Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9411544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9411618Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9411847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9411939Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9411947Z 
2025-09-07T08:14:18.9412035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9412220Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9412285Z     return mod(**inputs)
2025-09-07T08:14:18.9412520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9412590Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9412820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9412883Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9413142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9413209Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9413439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9413541Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9413768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9413842Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9414069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9414143Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9414146Z 
2025-09-07T08:14:18.9414218Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9414307Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9414496Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9414553Z     return mod(**inputs)
2025-09-07T08:14:18.9414787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9414851Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9415114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9415177Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9415420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9415492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9415718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9415796Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9416022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9416119Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9416359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9416430Z     return self.weight * hidden_states
2025-09-07T08:14:18.9416433Z 
2025-09-07T08:14:18.9416532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9416720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9416780Z     return mod(**inputs)
2025-09-07T08:14:18.9417020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9417084Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9417383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9417448Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9417675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9417748Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9417982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9418055Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9418282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9418356Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9418580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9418690Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9418693Z 
2025-09-07T08:14:18.9418784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9418966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9419027Z     return mod(**inputs)
2025-09-07T08:14:18.9419292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9419357Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9419591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9419656Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9419883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9419954Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9420185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9420255Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9420480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9420560Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9420820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9420894Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9420897Z 
2025-09-07T08:14:18.9421017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9421203Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9421269Z     return mod(**inputs)
2025-09-07T08:14:18.9421498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9421563Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9421790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9421850Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9422063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9422129Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9422363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9422432Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9422662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9422732Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9422956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9423081Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9423085Z 
2025-09-07T08:14:18.9423173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9423364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9423425Z     return mod(**inputs)
2025-09-07T08:14:18.9423652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9423721Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9423951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9424019Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9424229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9424332Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9424558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9424625Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9424892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9424964Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9425196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9425310Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9425313Z 
2025-09-07T08:14:18.9425404Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9425595Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9425654Z     return mod(**inputs)
2025-09-07T08:14:18.9425894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9425957Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9426218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9426285Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9426496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9426598Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9426827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9426899Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9427124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9427196Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9427422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9427530Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9427536Z 
2025-09-07T08:14:18.9427630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9427816Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9427874Z     return mod(**inputs)
2025-09-07T08:14:18.9428109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9428170Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9428403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9428464Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9428681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9428745Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9428972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9429046Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9429283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9429358Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9429583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9429726Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9429768Z 
2025-09-07T08:14:18.9429859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9430052Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9430116Z     return mod(**inputs)
2025-09-07T08:14:18.9430351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9430464Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9430698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9430759Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9430974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9431041Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9431278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9431350Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9431577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9431654Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9431916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9431996Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9431999Z 
2025-09-07T08:14:18.9432118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9432309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9432373Z     return mod(**inputs)
2025-09-07T08:14:18.9432607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9432679Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9432907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9432967Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9433185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9433254Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9433486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9433555Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9433785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9433858Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9434083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9434189Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9434192Z 
2025-09-07T08:14:18.9434282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9434471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9434532Z     return mod(**inputs)
2025-09-07T08:14:18.9434762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9434834Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9435065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9435135Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9435348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9435459Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9435688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9435756Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9435988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9436094Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9436326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9436417Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9436421Z 
2025-09-07T08:14:18.9436508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9436691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9436746Z     return mod(**inputs)
2025-09-07T08:14:18.9436978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9437038Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9437264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9437364Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9437588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9437662Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9437927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9438008Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9438239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9438314Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9438542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9438637Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9438640Z 
2025-09-07T08:14:18.9438739Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9438930Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9438991Z     return mod(**inputs)
2025-09-07T08:14:18.9439236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9439302Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9439536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9439601Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9439819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9439887Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9440113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9440187Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9440412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9440489Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9440719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9440815Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9440818Z 
2025-09-07T08:14:18.9440915Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9441136Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9441204Z     return mod(**inputs)
2025-09-07T08:14:18.9441437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9441502Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9441787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9441855Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9442075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9442142Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9442372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9442443Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9442669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9442741Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9442966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9443075Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9443079Z 
2025-09-07T08:14:18.9443153Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9443244Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9443465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9443524Z     return mod(**inputs)
2025-09-07T08:14:18.9443762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9443823Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9444054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9444116Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9444326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9444397Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9444623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9444709Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9444935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9445021Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9445251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9445320Z     return self.weight * hidden_states
2025-09-07T08:14:18.9445323Z 
2025-09-07T08:14:18.9445412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9445596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9445653Z     return mod(**inputs)
2025-09-07T08:14:18.9445890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9445952Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9446188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9446247Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9446457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9446524Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9446787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9446872Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9447098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9447449Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9447680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9447755Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9447759Z 
2025-09-07T08:14:18.9447855Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9448037Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9448099Z     return mod(**inputs)
2025-09-07T08:14:18.9448327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9448393Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9448623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9448684Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9448937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9449002Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9449262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9449342Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9449568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9449678Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9449905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9449983Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9449987Z 
2025-09-07T08:14:18.9450077Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9450265Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9450323Z     return mod(**inputs)
2025-09-07T08:14:18.9450562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9450633Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9450862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9450924Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9451138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9451207Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9451442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9455818Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9456122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9456243Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9456499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9456579Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9456583Z 
2025-09-07T08:14:18.9456686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9456895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9457024Z     return mod(**inputs)
2025-09-07T08:14:18.9457343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9457422Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9457706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9457779Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9458009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9458086Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9458328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9458415Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9458651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T08:14:18.9458773Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T08:14:18.9458777Z 
2025-09-07T08:14:18.9458854Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9458956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9459191Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9459254Z     return mod(**inputs)
2025-09-07T08:14:18.9459552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9459621Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9459857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9459921Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9460150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9460221Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9460454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9460531Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9460761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9460861Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9461091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9461164Z     return self.weight * hidden_states
2025-09-07T08:14:18.9461168Z 
2025-09-07T08:14:18.9461266Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9461467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9461527Z     return mod(**inputs)
2025-09-07T08:14:18.9461761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9461829Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9462062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9462128Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9462347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9462415Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9462648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9462720Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9462996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9463076Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9463306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9463415Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9463418Z 
2025-09-07T08:14:18.9463509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9463703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9463760Z     return mod(**inputs)
2025-09-07T08:14:18.9463990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9464057Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9464288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9464357Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9464569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9464639Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9464908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9464981Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9465247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9465322Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9465553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9465619Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9465624Z 
2025-09-07T08:14:18.9465712Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9465899Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9465955Z     return mod(**inputs)
2025-09-07T08:14:18.9466189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9466251Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9466482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9466550Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9466764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9466837Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9467064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9467133Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9467366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9467434Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9467663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9467784Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9467788Z 
2025-09-07T08:14:18.9467879Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9468063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9468118Z     return mod(**inputs)
2025-09-07T08:14:18.9468353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9468453Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9468690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9468756Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9468975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9469080Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9469308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9469389Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9469613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9469688Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9469913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9470028Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9470032Z 
2025-09-07T08:14:18.9470130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9470315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9470410Z     return mod(**inputs)
2025-09-07T08:14:18.9470645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9470741Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9470976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9471038Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9471255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9471322Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9471548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9471622Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9471848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9471925Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9472150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9472263Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9472267Z 
2025-09-07T08:14:18.9472354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9472540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9472610Z     return mod(**inputs)
2025-09-07T08:14:18.9472843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9472912Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9473140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9473205Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9473421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9473489Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9473719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9473786Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9474016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9474119Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9474346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9474499Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9474536Z 
2025-09-07T08:14:18.9474630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9474825Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9474882Z     return mod(**inputs)
2025-09-07T08:14:18.9475130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9475197Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9475427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9475494Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9475709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9475778Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9476015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9476141Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9476377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9476479Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9476715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9476784Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9476787Z 
2025-09-07T08:14:18.9476880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9477072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9477130Z     return mod(**inputs)
2025-09-07T08:14:18.9477364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9477430Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9477661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9477727Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9477948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9478022Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9478254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9478323Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9478553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9478624Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9478853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9478957Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9478961Z 
2025-09-07T08:14:18.9479058Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9479246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9479306Z     return mod(**inputs)
2025-09-07T08:14:18.9479539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9479602Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9479881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9479944Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9480154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9480257Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9480486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9480557Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9480786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9480865Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9481095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9481188Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9481192Z 
2025-09-07T08:14:18.9481286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9481468Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9481530Z     return mod(**inputs)
2025-09-07T08:14:18.9481794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9481861Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9482126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9482190Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9482410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9482478Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9482706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9482778Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9483004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9483085Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9483309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9483401Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9483407Z 
2025-09-07T08:14:18.9483498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9483682Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9483741Z     return mod(**inputs)
2025-09-07T08:14:18.9483975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9484044Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9484275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9484334Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9484554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9484620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9484854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9484924Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9485149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9485222Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9485483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9485579Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9485582Z 
2025-09-07T08:14:18.9485671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9485896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9485954Z     return mod(**inputs)
2025-09-07T08:14:18.9486187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9486250Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9486483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9486546Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9486755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9486821Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9487047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9487117Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9487386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9487458Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9487727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9487798Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9487802Z 
2025-09-07T08:14:18.9487878Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9487977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9488161Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9488224Z     return mod(**inputs)
2025-09-07T08:14:18.9488455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9488519Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9488757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9488818Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9489039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9489108Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9489334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9489411Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9489638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9489743Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9489968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9490044Z     return self.weight * hidden_states
2025-09-07T08:14:18.9490047Z 
2025-09-07T08:14:18.9490141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9490328Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9490387Z     return mod(**inputs)
2025-09-07T08:14:18.9490616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9490679Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9490907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9491003Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9491220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9491287Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9491552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9491623Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9491849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9491926Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9492150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9492223Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9492228Z 
2025-09-07T08:14:18.9492319Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9492504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9492559Z     return mod(**inputs)
2025-09-07T08:14:18.9492819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9492889Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9493117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9493212Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9493427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9493492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9493725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9493796Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9494024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9494097Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9494325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9494399Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9494402Z 
2025-09-07T08:14:18.9494498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9494690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9494747Z     return mod(**inputs)
2025-09-07T08:14:18.9494982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9495046Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9495277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9495344Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9495555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9495631Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9495859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9495931Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9496164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9496238Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9496473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9496630Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9496633Z 
2025-09-07T08:14:18.9496727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9496911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9497018Z     return mod(**inputs)
2025-09-07T08:14:18.9497293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9497356Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9497593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9497655Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9497868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9497939Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9498166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9498243Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9498509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9498584Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9498812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9498956Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9498960Z 
2025-09-07T08:14:18.9499054Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9499236Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9499298Z     return mod(**inputs)
2025-09-07T08:14:18.9499527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9499588Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9499821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9499885Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9500103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9500171Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9500395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9500467Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9500693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9500773Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9500996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9501103Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9501112Z 
2025-09-07T08:14:18.9501204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9501385Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9501447Z     return mod(**inputs)
2025-09-07T08:14:18.9501676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9501743Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9501974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9502097Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9502311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9502377Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9502611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9502715Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9502941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9503020Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9503411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9503565Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9503569Z 
2025-09-07T08:14:18.9503659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9503850Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9503906Z     return mod(**inputs)
2025-09-07T08:14:18.9504139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9504268Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9504508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9504574Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9504837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9504911Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9505146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9505218Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9505451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9505521Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9505747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9505819Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9505823Z 
2025-09-07T08:14:18.9505917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9506108Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9506164Z     return mod(**inputs)
2025-09-07T08:14:18.9506398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9506463Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9506692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9506757Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9506971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9507041Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9507267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9507334Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9507564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9507635Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9507860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9508006Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9508009Z 
2025-09-07T08:14:18.9508104Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9508292Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9508348Z     return mod(**inputs)
2025-09-07T08:14:18.9508634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9508702Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9508943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9509005Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9509218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9509292Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9509523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9509603Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9509833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9509946Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9510175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9510305Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9510309Z 
2025-09-07T08:14:18.9510406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9510594Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9510655Z     return mod(**inputs)
2025-09-07T08:14:18.9510888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9510952Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9511190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9511252Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9511475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9511543Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9511776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9511846Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9512072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9512145Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9512374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9512470Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9512473Z 
2025-09-07T08:14:18.9512560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9512762Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9512823Z     return mod(**inputs)
2025-09-07T08:14:18.9513054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9513122Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9513350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9513411Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9513624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9513736Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9513968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9514036Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9514305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9514378Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9514606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9514701Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9514705Z 
2025-09-07T08:14:18.9514792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9514977Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9515033Z     return mod(**inputs)
2025-09-07T08:14:18.9515264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9515327Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9515594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9515662Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9515905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9515971Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9516203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9516270Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9516497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9516571Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9516799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9516868Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9516874Z 
2025-09-07T08:14:18.9516947Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9517038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9517221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9517277Z     return mod(**inputs)
2025-09-07T08:14:18.9517504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9517565Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9517794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9517855Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9518068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9518132Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9518358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9518446Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9518678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9518770Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9518995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9519121Z     return self.weight * hidden_states
2025-09-07T08:14:18.9519125Z 
2025-09-07T08:14:18.9519215Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9519398Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9519457Z     return mod(**inputs)
2025-09-07T08:14:18.9519689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9519787Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9520018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9520080Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9520298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9520368Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9520598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9520678Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9520906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9521014Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9521276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9521359Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9521363Z 
2025-09-07T08:14:18.9521479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9521667Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9521722Z     return mod(**inputs)
2025-09-07T08:14:18.9521950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9522016Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9522244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9522306Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9522518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9522585Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9522813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9522887Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9523114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9523216Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9523444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9523515Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9523519Z 
2025-09-07T08:14:18.9523605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9523790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9523846Z     return mod(**inputs)
2025-09-07T08:14:18.9524075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9524137Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9524364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9524425Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9524634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9524881Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9525107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9525181Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9525446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9525546Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9525781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9525851Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9525854Z 
2025-09-07T08:14:18.9525930Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9526019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9526204Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9526267Z     return mod(**inputs)
2025-09-07T08:14:18.9526494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9526557Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9526819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9526880Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9527123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9527191Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9527418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9527490Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9527717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9527817Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9528042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9528113Z     return self.weight * hidden_states
2025-09-07T08:14:18.9528116Z 
2025-09-07T08:14:18.9528203Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9528386Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9528441Z     return mod(**inputs)
2025-09-07T08:14:18.9528668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9528731Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9528961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9529028Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9529243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9529310Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9529543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9529612Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9529842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9529912Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9530138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9530206Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9530243Z 
2025-09-07T08:14:18.9530329Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9530511Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9530565Z     return mod(**inputs)
2025-09-07T08:14:18.9530796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9530890Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9531118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9531182Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9531392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9531459Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9531685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9531755Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9531982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9532050Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9532309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9532377Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9532381Z 
2025-09-07T08:14:18.9532496Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9532679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9532733Z     return mod(**inputs)
2025-09-07T08:14:18.9532963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9533025Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9533256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9533315Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9533524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9533594Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9533818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9533889Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9534113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9534180Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9534414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9534537Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9534540Z 
2025-09-07T08:14:18.9534630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9534810Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9534869Z     return mod(**inputs)
2025-09-07T08:14:18.9535101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9535163Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9535395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9535453Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9535666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9535763Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9535990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9536060Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9536289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9536395Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9536629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9536753Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9536757Z 
2025-09-07T08:14:18.9536849Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9537037Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9537103Z     return mod(**inputs)
2025-09-07T08:14:18.9537403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9537471Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9537701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9537801Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9538021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9538091Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9538374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9538446Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9538672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9538748Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9538972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9539086Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9539090Z 
2025-09-07T08:14:18.9539180Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9539365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9539421Z     return mod(**inputs)
2025-09-07T08:14:18.9539650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9539714Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9539946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9540014Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9540227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9540296Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9540527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9540598Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9540830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9540903Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9541134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9541282Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9541286Z 
2025-09-07T08:14:18.9541374Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9541591Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9541646Z     return mod(**inputs)
2025-09-07T08:14:18.9541876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9541973Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9542203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9542266Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9542477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9542546Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9542777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9542848Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9543079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9543146Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9543407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9543477Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9543480Z 
2025-09-07T08:14:18.9543570Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9543783Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9543839Z     return mod(**inputs)
2025-09-07T08:14:18.9544069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9544134Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9544373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9544431Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9544640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9544710Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9544942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9545017Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9545242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9545309Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9545541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9545641Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9545645Z 
2025-09-07T08:14:18.9545738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9545919Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9545977Z     return mod(**inputs)
2025-09-07T08:14:18.9546209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9546270Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9546500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9546559Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9546771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9546834Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9547097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9547171Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9547396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9547516Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9547744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9547840Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9547846Z 
2025-09-07T08:14:18.9547941Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9548128Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9548187Z     return mod(**inputs)
2025-09-07T08:14:18.9548421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9548488Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9548717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9548780Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9549037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9549106Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9549376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9549450Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9549684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9549766Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9549992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9550090Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9550094Z 
2025-09-07T08:14:18.9550186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9550378Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9550435Z     return mod(**inputs)
2025-09-07T08:14:18.9550664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9550736Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9550965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9551030Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9551242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9551312Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9551541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9551610Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9551846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9551914Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9552140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9552235Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9552238Z 
2025-09-07T08:14:18.9552327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9552512Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9552601Z     return mod(**inputs)
2025-09-07T08:14:18.9552833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9552895Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9553123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9553225Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9553436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9553510Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9553737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9553805Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9554035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9554108Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9554339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9554408Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9554413Z 
2025-09-07T08:14:18.9554538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9554723Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9554779Z     return mod(**inputs)
2025-09-07T08:14:18.9555045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9555111Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9555343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9555409Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9555620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9555693Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9555922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9555999Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9556227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.9556351Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9556359Z 
2025-09-07T08:14:18.9556434Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9556524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9556709Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9556765Z     return mod(**inputs)
2025-09-07T08:14:18.9556994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9557056Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9557283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9557347Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9557556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9557625Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9557851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9557920Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9558149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9558280Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9558511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9558579Z     return self.weight * hidden_states
2025-09-07T08:14:18.9558613Z 
2025-09-07T08:14:18.9558704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9558893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9558951Z     return mod(**inputs)
2025-09-07T08:14:18.9559187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9559252Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9559486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9559551Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9559763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9559840Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9560112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9560190Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9560418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9560524Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9560757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9560825Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9560828Z 
2025-09-07T08:14:18.9560924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9561105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9561161Z     return mod(**inputs)
2025-09-07T08:14:18.9561394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9561465Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9561695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9561756Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9561968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9562033Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9562257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9562329Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9562553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9562625Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9562850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9562918Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9562921Z 
2025-09-07T08:14:18.9563010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9563192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9563249Z     return mod(**inputs)
2025-09-07T08:14:18.9563476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9563537Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9563809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9563872Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9564084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9564185Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9564415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9564484Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9564710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9564782Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9565006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9565127Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9565130Z 
2025-09-07T08:14:18.9565219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9565399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9565455Z     return mod(**inputs)
2025-09-07T08:14:18.9565725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9565790Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9566051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9566115Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9566326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9566390Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9566622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9566690Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9566916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9566989Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9567214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9567328Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9567332Z 
2025-09-07T08:14:18.9567420Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9567602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9567657Z     return mod(**inputs)
2025-09-07T08:14:18.9567885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9567949Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9568175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9568237Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9568450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9568517Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9568743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9568811Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9569038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9569109Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9569382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9569489Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9569492Z 
2025-09-07T08:14:18.9569581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9569804Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9569860Z     return mod(**inputs)
2025-09-07T08:14:18.9570092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9570153Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9570381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9570443Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9570655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9570721Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9570946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9571017Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9571277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9571353Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9571611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9571755Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9571759Z 
2025-09-07T08:14:18.9571848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9572031Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9572085Z     return mod(**inputs)
2025-09-07T08:14:18.9572314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9572375Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9572608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9572667Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9572879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9572944Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9573173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9573245Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9573471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9573543Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9573769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9573839Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9573843Z 
2025-09-07T08:14:18.9573931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9574113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9574170Z     return mod(**inputs)
2025-09-07T08:14:18.9574397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9574458Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9574688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9574787Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9575000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9575065Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9575328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9575397Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9575623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9575696Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9575921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9576019Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9576023Z 
2025-09-07T08:14:18.9576111Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9576293Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9576349Z     return mod(**inputs)
2025-09-07T08:14:18.9576611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9576677Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9576938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9577002Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9577272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9577338Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9577565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9577635Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9577862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9577931Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9578157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9578251Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9578255Z 
2025-09-07T08:14:18.9578344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9578526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9578581Z     return mod(**inputs)
2025-09-07T08:14:18.9578809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9578873Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9579101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9579162Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9579374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9579441Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9579669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9579737Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9579966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9580036Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9580262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9580388Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9580391Z 
2025-09-07T08:14:18.9580478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9580663Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9580767Z     return mod(**inputs)
2025-09-07T08:14:18.9581001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9581065Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9581292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9581354Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9581572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9581644Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9581876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9581945Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9582218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9582291Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9582550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9582640Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9582643Z 
2025-09-07T08:14:18.9582733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9582913Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9582969Z     return mod(**inputs)
2025-09-07T08:14:18.9583200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9583261Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9583494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9583555Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9583765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9583833Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9584058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9584129Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9584352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9584425Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9584650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9584715Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9584719Z 
2025-09-07T08:14:18.9584793Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9584880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9585060Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9585116Z     return mod(**inputs)
2025-09-07T08:14:18.9585344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9585406Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9585633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9585731Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9585941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9586006Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9586238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9586353Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9586583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9586668Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9586894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9586959Z     return self.weight * hidden_states
2025-09-07T08:14:18.9586963Z 
2025-09-07T08:14:18.9587050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9587235Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9587290Z     return mod(**inputs)
2025-09-07T08:14:18.9587519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9587618Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9587849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9587984Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9588201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9588270Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9588494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9588575Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9588801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9588909Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9589137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9589206Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9589210Z 
2025-09-07T08:14:18.9589300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9589481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9589536Z     return mod(**inputs)
2025-09-07T08:14:18.9589768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9589830Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9590063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9590122Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9590331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9590402Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9590628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9590706Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9590929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9591032Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9591258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9591366Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9591369Z 
2025-09-07T08:14:18.9591458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9591637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9591729Z     return mod(**inputs)
2025-09-07T08:14:18.9591961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9592022Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9592251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9592312Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9592524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9592592Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9592816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9592893Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9593149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9593253Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9593515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9593586Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9593589Z 
2025-09-07T08:14:18.9593657Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9593745Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9593927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9593983Z     return mod(**inputs)
2025-09-07T08:14:18.9594213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9594272Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9594500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9594564Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9594774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9594842Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9595066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9595135Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9595362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9595457Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9595684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9595749Z     return self.weight * hidden_states
2025-09-07T08:14:18.9595753Z 
2025-09-07T08:14:18.9595844Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9596024Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9596079Z     return mod(**inputs)
2025-09-07T08:14:18.9596311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9596370Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9596601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9596694Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9596910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9596978Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9597207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9597311Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9597537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9597611Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9597836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9597903Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9597906Z 
2025-09-07T08:14:18.9597997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9598177Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9598233Z     return mod(**inputs)
2025-09-07T08:14:18.9598460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9598556Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9598789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9598849Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9599090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9599155Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9599380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9599453Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9599678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9599750Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9599974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9600044Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9600047Z 
2025-09-07T08:14:18.9600134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9600316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9600374Z     return mod(**inputs)
2025-09-07T08:14:18.9600602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9600665Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9600895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9600954Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9601174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9601241Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9601469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9601536Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9601763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9601840Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9602066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9602253Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9602256Z 
2025-09-07T08:14:18.9602346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9602530Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9602585Z     return mod(**inputs)
2025-09-07T08:14:18.9602855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9602925Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9603303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9603366Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9603581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9603651Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9603881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9603952Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9604184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9604325Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9604558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9604721Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9604724Z 
2025-09-07T08:14:18.9604817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9605001Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9605057Z     return mod(**inputs)
2025-09-07T08:14:18.9605294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9605356Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9605584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9605652Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9605864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9605935Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9606164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9606237Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9606463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9606534Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9606767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9606876Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9606879Z 
2025-09-07T08:14:18.9606969Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9607154Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9607212Z     return mod(**inputs)
2025-09-07T08:14:18.9607449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9607511Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9607742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9607805Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9608074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9608146Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9608378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9608451Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9608727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9608800Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9609028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9609173Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9609176Z 
2025-09-07T08:14:18.9609271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9609459Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9609517Z     return mod(**inputs)
2025-09-07T08:14:18.9609748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9609811Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9610080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9610143Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9610392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9610461Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9610691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9610763Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9610991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9611062Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9611286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9611360Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9611363Z 
2025-09-07T08:14:18.9611451Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9611635Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9611694Z     return mod(**inputs)
2025-09-07T08:14:18.9611923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9611985Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9612211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9612273Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9612488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9612552Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9612783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9612850Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9613075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9613146Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9613374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9613472Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9613513Z 
2025-09-07T08:14:18.9613602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9613786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9613841Z     return mod(**inputs)
2025-09-07T08:14:18.9614077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9614173Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9614404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9614470Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9614683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9614751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9614984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9615054Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9615281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9615349Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9615605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9615702Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9615706Z 
2025-09-07T08:14:18.9615822Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9616008Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9616062Z     return mod(**inputs)
2025-09-07T08:14:18.9616304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9616366Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9616594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9616656Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9616875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9616946Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9617218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9617288Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9617519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9617589Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9617816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9617907Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9617911Z 
2025-09-07T08:14:18.9618003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9618189Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9618248Z     return mod(**inputs)
2025-09-07T08:14:18.9618478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9618542Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9618773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9618833Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9619044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9619157Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9619385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9619456Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9619682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9619792Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9620025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9620116Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9620119Z 
2025-09-07T08:14:18.9620211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9620392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9620450Z     return mod(**inputs)
2025-09-07T08:14:18.9620679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9620739Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9620971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9621067Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9621282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9621399Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9621629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9621704Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9621930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9622003Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9622227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9622293Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9622299Z 
2025-09-07T08:14:18.9622370Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9622459Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9622643Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9622697Z     return mod(**inputs)
2025-09-07T08:14:18.9622929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9622989Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9623218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9623282Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9623492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9623559Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9623785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9623855Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9624081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9624176Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9624406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9624470Z     return self.weight * hidden_states
2025-09-07T08:14:18.9624473Z 
2025-09-07T08:14:18.9624613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9624795Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9624849Z     return mod(**inputs)
2025-09-07T08:14:18.9625081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9625178Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9625414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9625475Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9625687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9625753Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9625978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9626058Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9626282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9626354Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9626616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9626685Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9626689Z 
2025-09-07T08:14:18.9626779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9626996Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9627057Z     return mod(**inputs)
2025-09-07T08:14:18.9627290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9627351Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9627595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9627655Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9627872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9627943Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9628170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9628248Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9628475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9628549Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9628774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9628843Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9628849Z 
2025-09-07T08:14:18.9628937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9629116Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9629172Z     return mod(**inputs)
2025-09-07T08:14:18.9629402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9629466Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9629695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9629754Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9629965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9630029Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9630300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9630369Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9630594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9630703Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9630930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9631052Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9631055Z 
2025-09-07T08:14:18.9631143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9631324Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9631381Z     return mod(**inputs)
2025-09-07T08:14:18.9631608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9631676Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9631904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9631968Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9632214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9632280Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9632539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9632610Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9632838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9632908Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9633136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9633247Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9633250Z 
2025-09-07T08:14:18.9633338Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9633524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9633579Z     return mod(**inputs)
2025-09-07T08:14:18.9633813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9633876Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9634103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9634167Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9634381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9634452Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9634678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9634746Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9634975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9635046Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9635275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9635381Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9635384Z 
2025-09-07T08:14:18.9635474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9635657Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9635753Z     return mod(**inputs)
2025-09-07T08:14:18.9635982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9636044Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9636313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9636374Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9636585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9636653Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9636879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9636949Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9637176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9637246Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9637472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9637652Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9637656Z 
2025-09-07T08:14:18.9637746Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9637965Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9638023Z     return mod(**inputs)
2025-09-07T08:14:18.9638252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9638313Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9638542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9638604Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9638814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9638878Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9639106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9639176Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9639401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9639474Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9639697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9639765Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9639769Z 
2025-09-07T08:14:18.9639862Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9640042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9640100Z     return mod(**inputs)
2025-09-07T08:14:18.9640328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9640393Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9640620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9640681Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9640894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9640958Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9641186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9641289Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9641515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9641587Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9641850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9641948Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9641952Z 
2025-09-07T08:14:18.9642039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9642219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9642277Z     return mod(**inputs)
2025-09-07T08:14:18.9642503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9642568Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9642795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9642857Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9643113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9643180Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9643439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9643508Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9643738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9643815Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9644043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9644139Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9644143Z 
2025-09-07T08:14:18.9644229Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9644410Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9644467Z     return mod(**inputs)
2025-09-07T08:14:18.9644695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9644759Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9644986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9645050Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9645259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9645327Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9645550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9645618Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9645845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9645917Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9646147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9646235Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9646238Z 
2025-09-07T08:14:18.9646325Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9646507Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9646596Z     return mod(**inputs)
2025-09-07T08:14:18.9646827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9646888Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9647118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9647214Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9647425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9647494Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9647721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9647793Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9648018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9648089Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9648315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9648404Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9648413Z 
2025-09-07T08:14:18.9648533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9648713Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9648768Z     return mod(**inputs)
2025-09-07T08:14:18.9649029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9649092Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9649322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9649384Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9649597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9649661Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9649886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9649961Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9650184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9650257Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9650480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9650544Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9650548Z 
2025-09-07T08:14:18.9650638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9650822Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9650879Z     return mod(**inputs)
2025-09-07T08:14:18.9651108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9651172Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9651405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9651465Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9651680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9651745Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9651971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9652039Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9652302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T08:14:18.9652432Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9652435Z 
2025-09-07T08:14:18.9652508Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9652638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9652821Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9652877Z     return mod(**inputs)
2025-09-07T08:14:18.9653115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9653178Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9653410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9653473Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9653685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9653757Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9653987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9654109Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9654339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9654463Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9654695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9654765Z     return self.weight * hidden_states
2025-09-07T08:14:18.9654768Z 
2025-09-07T08:14:18.9654863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9655046Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9655109Z     return mod(**inputs)
2025-09-07T08:14:18.9655340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9655408Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9655640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9655702Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9655913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9655982Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9656211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9656289Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9656517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9656629Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9656854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9656927Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9656930Z 
2025-09-07T08:14:18.9657019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9657250Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9657308Z     return mod(**inputs)
2025-09-07T08:14:18.9657537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9657602Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9657872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9657934Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9658145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9658248Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9658476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9658550Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9658778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9658878Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9659103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9659177Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9659180Z 
2025-09-07T08:14:18.9659266Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9659447Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9659502Z     return mod(**inputs)
2025-09-07T08:14:18.9659763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9659827Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9660088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9660153Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9660363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9660432Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9660665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9660738Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9660965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9661065Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9661291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9661359Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9661362Z 
2025-09-07T08:14:18.9661431Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9661521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9661703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9661761Z     return mod(**inputs)
2025-09-07T08:14:18.9661989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9662050Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9662281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9662347Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9662560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9662624Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9662850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9662922Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9663146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9663275Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9663499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9663567Z     return self.weight * hidden_states
2025-09-07T08:14:18.9663570Z 
2025-09-07T08:14:18.9663705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9663887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9663945Z     return mod(**inputs)
2025-09-07T08:14:18.9664175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9664239Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9664467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9664528Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9664746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9664812Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9665039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9665143Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9665374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9665476Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9665705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9665774Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9665778Z 
2025-09-07T08:14:18.9665869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9666058Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9666114Z     return mod(**inputs)
2025-09-07T08:14:18.9666355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9666424Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9666655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9666725Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9666935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9667004Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9667232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9671560Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9671866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9671948Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9672200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9672279Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9672284Z 
2025-09-07T08:14:18.9672387Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9672599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9672663Z     return mod(**inputs)
2025-09-07T08:14:18.9672914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9672989Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9673232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9673368Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9673603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9673680Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9673968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9674045Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9674288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9674363Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9674596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9674719Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9674725Z 
2025-09-07T08:14:18.9674827Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9675030Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9675090Z     return mod(**inputs)
2025-09-07T08:14:18.9675380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9675449Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9675713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9675784Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9676003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9676085Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9676319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9676398Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9676624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9676698Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9676929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9677043Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9677047Z 
2025-09-07T08:14:18.9677147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9677333Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9677394Z     return mod(**inputs)
2025-09-07T08:14:18.9677644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9677714Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9677965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9678026Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9678246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9678319Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9678552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9678628Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9678854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9678936Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9679203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9679319Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9679323Z 
2025-09-07T08:14:18.9679423Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9679618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9679725Z     return mod(**inputs)
2025-09-07T08:14:18.9679970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9680039Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9680284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9680346Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9680567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9680638Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9680870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9680944Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9681246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9681324Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9681585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9681738Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9681742Z 
2025-09-07T08:14:18.9681837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9682033Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9682100Z     return mod(**inputs)
2025-09-07T08:14:18.9682334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9682406Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9682639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9682704Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9682923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9682993Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9683229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9683302Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9683536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9683616Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9683844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9683924Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9683929Z 
2025-09-07T08:14:18.9684025Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9684214Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9684273Z     return mod(**inputs)
2025-09-07T08:14:18.9684507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9684575Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9684811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9684937Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9685157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9685236Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9685473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9685578Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9685811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9685884Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9686117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9686219Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9686222Z 
2025-09-07T08:14:18.9686321Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9686518Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9686580Z     return mod(**inputs)
2025-09-07T08:14:18.9686819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9686920Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9687158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9687224Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9687468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9687540Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9687768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9687841Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9688065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9688136Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9688364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9688456Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9688459Z 
2025-09-07T08:14:18.9688551Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9688735Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9688791Z     return mod(**inputs)
2025-09-07T08:14:18.9689022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9689084Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9689318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9689380Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9689592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9689664Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9689893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9689966Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9690191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9690262Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9690486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9690627Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9690631Z 
2025-09-07T08:14:18.9690721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9690905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9690998Z     return mod(**inputs)
2025-09-07T08:14:18.9691233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9691301Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9691540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9691600Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9691812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9691878Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9692112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9692180Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9692406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9692514Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9692748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9692882Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9692885Z 
2025-09-07T08:14:18.9692975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9693159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9693223Z     return mod(**inputs)
2025-09-07T08:14:18.9693456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9693521Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9693752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9693815Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9694036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9694105Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9694340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9694413Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9694644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9694716Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9694949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9695024Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9695028Z 
2025-09-07T08:14:18.9695101Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9695201Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9695385Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9695443Z     return mod(**inputs)
2025-09-07T08:14:18.9695682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9695748Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9695986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9696046Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9696302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9696380Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9696606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9696720Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9696949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9697056Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9697354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9697424Z     return self.weight * hidden_states
2025-09-07T08:14:18.9697427Z 
2025-09-07T08:14:18.9697529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9697714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9697778Z     return mod(**inputs)
2025-09-07T08:14:18.9698015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9698079Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9698356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9698422Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9698678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9698749Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9698982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9699059Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9699288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9699373Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9699602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9699681Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9699684Z 
2025-09-07T08:14:18.9699777Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9699962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9700026Z     return mod(**inputs)
2025-09-07T08:14:18.9700256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9700322Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9700549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9700611Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9700828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9700893Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9701126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9701196Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9701422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9701497Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9701720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9701792Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9701827Z 
2025-09-07T08:14:18.9701917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9702101Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9702155Z     return mod(**inputs)
2025-09-07T08:14:18.9702386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9702483Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9702713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9702776Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9702986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9703220Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9703455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9703526Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9703763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9703835Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9704151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9704275Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9704278Z 
2025-09-07T08:14:18.9704415Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9704603Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9704658Z     return mod(**inputs)
2025-09-07T08:14:18.9704890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9704954Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9705185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9705252Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9705463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9705532Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9705758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9705828Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9706058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9706128Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9706360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9706469Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9706473Z 
2025-09-07T08:14:18.9706566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9706750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9706808Z     return mod(**inputs)
2025-09-07T08:14:18.9707041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9707102Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9707335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9707395Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9707606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9707723Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9707951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9708025Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9708302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9708377Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9708609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9708723Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9708726Z 
2025-09-07T08:14:18.9708825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9709009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9709073Z     return mod(**inputs)
2025-09-07T08:14:18.9709307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9709369Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9709636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9709701Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9709920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9710039Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9710272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9710345Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9710578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9710658Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9710882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9711031Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9711037Z 
2025-09-07T08:14:18.9711127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9711312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9711373Z     return mod(**inputs)
2025-09-07T08:14:18.9711603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9711667Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9711893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9711955Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9712169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9712233Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9712464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9712534Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9712761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9712835Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9713058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9713128Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9713132Z 
2025-09-07T08:14:18.9713254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9713437Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9713493Z     return mod(**inputs)
2025-09-07T08:14:18.9713721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9713820Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9714051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9714115Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9714325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9714390Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9714621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9714690Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9714916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9714986Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9715241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9715345Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9715348Z 
2025-09-07T08:14:18.9715435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9715650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9715706Z     return mod(**inputs)
2025-09-07T08:14:18.9715938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9716001Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9716230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9716293Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9716504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9716574Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9716802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9716872Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9717099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9717170Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9717398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9717488Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9717491Z 
2025-09-07T08:14:18.9717581Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9717763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9717820Z     return mod(**inputs)
2025-09-07T08:14:18.9718053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9718114Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9718343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9718402Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9718612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9718679Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9718940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9719012Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9719238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9719346Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9719582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9719671Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9719674Z 
2025-09-07T08:14:18.9719767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9719946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9720003Z     return mod(**inputs)
2025-09-07T08:14:18.9720231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9720293Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9720524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9720587Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9720836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9720901Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9721157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9721234Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9721458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9721533Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9721757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9721852Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9721855Z 
2025-09-07T08:14:18.9721942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9722130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9722192Z     return mod(**inputs)
2025-09-07T08:14:18.9722422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9722486Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9722714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9722773Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9722990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9723055Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9723289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9723359Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9723585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9723662Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9723886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9723959Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9723962Z 
2025-09-07T08:14:18.9724034Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9724130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9724355Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9724419Z     return mod(**inputs)
2025-09-07T08:14:18.9724659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9724762Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9724998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9725061Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9725272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9725339Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9725565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9725650Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9725877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9725968Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9726232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9726302Z     return self.weight * hidden_states
2025-09-07T08:14:18.9726305Z 
2025-09-07T08:14:18.9726396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9726610Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9726669Z     return mod(**inputs)
2025-09-07T08:14:18.9726898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9726960Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9727192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9727252Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9727465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9727532Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9727760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9727836Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9728064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9728175Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9728400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9728476Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9728480Z 
2025-09-07T08:14:18.9728568Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9728751Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9728809Z     return mod(**inputs)
2025-09-07T08:14:18.9729045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9729110Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9729342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9729402Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9729615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9729681Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9729963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9730039Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9730271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9730415Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9730642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9730717Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9730721Z 
2025-09-07T08:14:18.9730808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9730991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9731046Z     return mod(**inputs)
2025-09-07T08:14:18.9731274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9731339Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9731569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9731632Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9731897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9731966Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9732225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9732300Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9732529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9732628Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9732856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9732924Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9732927Z 
2025-09-07T08:14:18.9733014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9733200Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9733255Z     return mod(**inputs)
2025-09-07T08:14:18.9733485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9733546Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9733773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9733834Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9734043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9734113Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9734337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9734413Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9734640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T08:14:18.9734757Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T08:14:18.9734762Z 
2025-09-07T08:14:18.9734833Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9734921Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9735105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9735159Z     return mod(**inputs)
2025-09-07T08:14:18.9735425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9735492Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9735720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9735822Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9736034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9736099Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9736328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9736399Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9736627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9736727Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9736957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9737023Z     return self.weight * hidden_states
2025-09-07T08:14:18.9737026Z 
2025-09-07T08:14:18.9737116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9737396Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9737454Z     return mod(**inputs)
2025-09-07T08:14:18.9737721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9737785Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9738022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9738087Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9738301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9738372Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9738601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9738677Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9738908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9738986Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9739214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9739281Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9739284Z 
2025-09-07T08:14:18.9739374Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9739556Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9739613Z     return mod(**inputs)
2025-09-07T08:14:18.9739842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9739902Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9740135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9740196Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9740406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9740473Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9740699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9740770Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9740993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9741100Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9741326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9741392Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9741628Z 
2025-09-07T08:14:18.9741719Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9741900Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9741959Z     return mod(**inputs)
2025-09-07T08:14:18.9742188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9742249Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9742481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9742542Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9742753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9742818Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9743075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9743149Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9743406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9743480Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9743704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9743824Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9743829Z 
2025-09-07T08:14:18.9743917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9744096Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9744154Z     return mod(**inputs)
2025-09-07T08:14:18.9744381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9744448Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9744674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9744734Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9744946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9745011Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9745240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9745308Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9745531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9745602Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9745827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9745940Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9745943Z 
2025-09-07T08:14:18.9746031Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9746215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9746270Z     return mod(**inputs)
2025-09-07T08:14:18.9746497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9746595Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9746822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9746884Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9747095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9747194Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9747422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9747492Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9747719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9747787Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9748013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9748122Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9748126Z 
2025-09-07T08:14:18.9748213Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9748396Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9748485Z     return mod(**inputs)
2025-09-07T08:14:18.9748722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9748784Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9749045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9749108Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9749319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9749389Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9749613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9749680Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9749908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9749978Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9750204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9750349Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9750353Z 
2025-09-07T08:14:18.9750443Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9750625Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9750682Z     return mod(**inputs)
2025-09-07T08:14:18.9750914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9750974Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9751204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9751267Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9751477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9751548Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9751774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9751844Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9752067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9752174Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9752400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9752467Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9752470Z 
2025-09-07T08:14:18.9752609Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9752790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9752848Z     return mod(**inputs)
2025-09-07T08:14:18.9753079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9753140Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9753373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9753432Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9753646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9753711Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9753936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9754040Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9754267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9754369Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9754596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9754697Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9754700Z 
2025-09-07T08:14:18.9754791Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9754973Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9755030Z     return mod(**inputs)
2025-09-07T08:14:18.9755258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9755320Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9755551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9755613Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9755829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9755894Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9756122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9756191Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9756416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9756488Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9756711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9756807Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9756810Z 
2025-09-07T08:14:18.9756898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9757081Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9757136Z     return mod(**inputs)
2025-09-07T08:14:18.9757382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9757444Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9757709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9757772Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9757983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9758048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9758311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9758379Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9758611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9758679Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9758904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9758998Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9759002Z 
2025-09-07T08:14:18.9759087Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9759270Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9759324Z     return mod(**inputs)
2025-09-07T08:14:18.9759591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9759653Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9759911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9759974Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9760183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9760251Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9760479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9760548Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9760775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9760846Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9761075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9761169Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9761173Z 
2025-09-07T08:14:18.9761263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9761443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9761498Z     return mod(**inputs)
2025-09-07T08:14:18.9761728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9761790Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9762022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9762081Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9762294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9762363Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9762590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9762660Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9762883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9762951Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9763216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9763284Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9763287Z 
2025-09-07T08:14:18.9763361Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9763449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9763670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9763726Z     return mod(**inputs)
2025-09-07T08:14:18.9763959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9764022Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9764253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9764314Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9764524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9764591Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9764818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9764887Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9765146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9765243Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9765508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9765579Z     return self.weight * hidden_states
2025-09-07T08:14:18.9765582Z 
2025-09-07T08:14:18.9765671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9765858Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9765916Z     return mod(**inputs)
2025-09-07T08:14:18.9766147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9766208Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9766443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9766507Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9766717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9766785Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9767013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9767084Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9767313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9767385Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9767612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9767681Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9767686Z 
2025-09-07T08:14:18.9767777Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9767958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9768013Z     return mod(**inputs)
2025-09-07T08:14:18.9768245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9768306Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9768536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9768632Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9768844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9768912Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9769138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9769270Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9769497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9769569Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9769795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9769860Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9769863Z 
2025-09-07T08:14:18.9769953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9770133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9770191Z     return mod(**inputs)
2025-09-07T08:14:18.9770421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9770518Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9770752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9770844Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9771061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9771126Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9771351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9771424Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9771651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9771723Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9771948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9772068Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9772074Z 
2025-09-07T08:14:18.9772164Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9772345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9772401Z     return mod(**inputs)
2025-09-07T08:14:18.9772629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9772697Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9772925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9772986Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9773199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9773266Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9773494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9773564Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9773788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9773861Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9774085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9774256Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9774259Z 
2025-09-07T08:14:18.9774346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9774530Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9774619Z     return mod(**inputs)
2025-09-07T08:14:18.9774849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9774914Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9775143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9775207Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9775418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9775487Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9775715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9775783Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9776053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9776127Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9776352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9776492Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9776496Z 
2025-09-07T08:14:18.9776584Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9776766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9776824Z     return mod(**inputs)
2025-09-07T08:14:18.9777069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9777132Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9777421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9777494Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9777709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9777784Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9778016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9778089Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9778320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9778393Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9778627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9778773Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9778778Z 
2025-09-07T08:14:18.9778878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9779068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9779126Z     return mod(**inputs)
2025-09-07T08:14:18.9779365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9779427Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9779663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9779767Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9779982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9780054Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9780282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9780406Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9780632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9780709Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9780939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9781009Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9781012Z 
2025-09-07T08:14:18.9781107Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9781290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9781352Z     return mod(**inputs)
2025-09-07T08:14:18.9781585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9781651Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9781923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9781986Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9782247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9782314Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9782545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9782618Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9782848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9782926Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9783156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9783258Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9783265Z 
2025-09-07T08:14:18.9783358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9783544Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9783609Z     return mod(**inputs)
2025-09-07T08:14:18.9783840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9783906Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9784138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9784199Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9784415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9784483Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9784718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9784788Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9785017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9785096Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9785321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9785457Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9785460Z 
2025-09-07T08:14:18.9785549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9785732Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9785793Z     return mod(**inputs)
2025-09-07T08:14:18.9786060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9786130Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9786360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9786428Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9786641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9786708Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9786940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9787011Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9787244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9787351Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9787582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9787682Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9787716Z 
2025-09-07T08:14:18.9787806Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9787993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9788048Z     return mod(**inputs)
2025-09-07T08:14:18.9788282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9788353Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9788582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9788648Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9788866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9788938Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9789164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9789236Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9789467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9789539Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9789773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9789862Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9789865Z 
2025-09-07T08:14:18.9789953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9790141Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9790197Z     return mod(**inputs)
2025-09-07T08:14:18.9790437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9790498Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9790733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9790795Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9791013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9791120Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9791350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9791426Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9791689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9791763Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9791997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9792064Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9792067Z 
2025-09-07T08:14:18.9792147Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9792236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9792421Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9792482Z     return mod(**inputs)
2025-09-07T08:14:18.9792712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9792779Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9793048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9793117Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9793383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9793451Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9793685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9793767Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9794003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9794090Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9794317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9794395Z     return self.weight * hidden_states
2025-09-07T08:14:18.9794398Z 
2025-09-07T08:14:18.9794487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9794674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9794730Z     return mod(**inputs)
2025-09-07T08:14:18.9794961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9795030Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9795259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9795326Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9795535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9795608Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9795840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9795919Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9796151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9796260Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9796493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9796565Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9796605Z 
2025-09-07T08:14:18.9796694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9796883Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9796941Z     return mod(**inputs)
2025-09-07T08:14:18.9797179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9797275Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9797509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9797577Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9797789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9797860Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9798088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9798173Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9798399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9798500Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9798767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9798844Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9798847Z 
2025-09-07T08:14:18.9798971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9799157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9799217Z     return mod(**inputs)
2025-09-07T08:14:18.9799453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9799519Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9799752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9799813Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9800033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9800102Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9800330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9800411Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9800640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9800746Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9800974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9801042Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9801045Z 
2025-09-07T08:14:18.9801122Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9801210Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9801402Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9801458Z     return mod(**inputs)
2025-09-07T08:14:18.9801687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9801751Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9801978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9802039Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9802288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9802357Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9802582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9802653Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9802918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9803014Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9803397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9803465Z     return self.weight * hidden_states
2025-09-07T08:14:18.9803468Z 
2025-09-07T08:14:18.9803557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9803742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9803800Z     return mod(**inputs)
2025-09-07T08:14:18.9804031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9804093Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9804410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9804474Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9804727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9804796Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9805022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9805094Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9805320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9805392Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9805619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9805687Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9805692Z 
2025-09-07T08:14:18.9805784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9805964Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9806020Z     return mod(**inputs)
2025-09-07T08:14:18.9806250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9806310Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9806541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9806602Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9806812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9806879Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9807106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9807177Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9807402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9807473Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9807697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9807762Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9807766Z 
2025-09-07T08:14:18.9807904Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9808085Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9808142Z     return mod(**inputs)
2025-09-07T08:14:18.9808368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9808478Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9808711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9808773Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9808985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9809048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9809274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9809345Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9809570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9809640Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9809897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9810019Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9810022Z 
2025-09-07T08:14:18.9810139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9810321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9810378Z     return mod(**inputs)
2025-09-07T08:14:18.9810606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9810670Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9810899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9810959Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9811171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9811238Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9811465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9811534Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9811759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9811828Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9812052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9812165Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9812169Z 
2025-09-07T08:14:18.9812255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9812438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9812496Z     return mod(**inputs)
2025-09-07T08:14:18.9812723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9812786Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9813016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9813077Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9813286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9813390Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9813622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9813689Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9813918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9814021Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9814250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9814357Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9814361Z 
2025-09-07T08:14:18.9814449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9814633Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9814688Z     return mod(**inputs)
2025-09-07T08:14:18.9814921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9814981Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9815208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9815319Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9815532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9815600Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9815858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9815929Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9816155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9816225Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9816451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9816595Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9816599Z 
2025-09-07T08:14:18.9816691Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9816871Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9816926Z     return mod(**inputs)
2025-09-07T08:14:18.9817210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9817272Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9817504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9817565Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9817776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9817844Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9818068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9818142Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9818364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9818435Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9818660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9818727Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9818731Z 
2025-09-07T08:14:18.9818820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9819041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9819098Z     return mod(**inputs)
2025-09-07T08:14:18.9819325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9819385Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9819653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9819714Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9819929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9819993Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9820227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9820296Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9820522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9820592Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9820816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9820949Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9820953Z 
2025-09-07T08:14:18.9821043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9821258Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9821316Z     return mod(**inputs)
2025-09-07T08:14:18.9821546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9821610Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9821838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9821901Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9822113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9822178Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9822411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9822480Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9822707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9822777Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9823002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9823094Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9823101Z 
2025-09-07T08:14:18.9823188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9823375Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9823429Z     return mod(**inputs)
2025-09-07T08:14:18.9823660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9823724Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9823952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9824014Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9824225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9824291Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9824528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9824634Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9824870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9824941Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9825213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9825305Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9825308Z 
2025-09-07T08:14:18.9825399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9825587Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9825644Z     return mod(**inputs)
2025-09-07T08:14:18.9825881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9825946Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9826175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9826244Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9826486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9826562Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9826821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9826893Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9827129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9827200Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9827431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9827521Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9827524Z 
2025-09-07T08:14:18.9827622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9827807Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9827864Z     return mod(**inputs)
2025-09-07T08:14:18.9828101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9828168Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9828403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9828463Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9828676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9828751Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9828978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9829053Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9829285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9829363Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9829590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9829657Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9829660Z 
2025-09-07T08:14:18.9829755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9829938Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9830034Z     return mod(**inputs)
2025-09-07T08:14:18.9830264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9830327Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9830562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9830661Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9830879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9830948Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9831171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9831242Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9831467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T08:14:18.9831593Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9831596Z 
2025-09-07T08:14:18.9831665Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9831754Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9831968Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9832028Z     return mod(**inputs)
2025-09-07T08:14:18.9832263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9832359Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9832599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9832663Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9832873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9832943Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9833167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9833239Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9833464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9833558Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9833787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9833854Z     return self.weight * hidden_states
2025-09-07T08:14:18.9833857Z 
2025-09-07T08:14:18.9833948Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9834129Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9834187Z     return mod(**inputs)
2025-09-07T08:14:18.9834416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9834476Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9834707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9834767Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9834977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9835042Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9835267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9835338Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9835568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9835703Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9835930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9835999Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9836059Z 
2025-09-07T08:14:18.9836154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9836343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9836403Z     return mod(**inputs)
2025-09-07T08:14:18.9836633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9836697Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9836927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9836988Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9837200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9837271Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9837511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9837624Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9837856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9837967Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9838196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9838277Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9838280Z 
2025-09-07T08:14:18.9838373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9838564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9838621Z     return mod(**inputs)
2025-09-07T08:14:18.9838851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9838919Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9839151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9839218Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9839433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9839500Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9839728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9839800Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9840028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9840099Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9840324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9840453Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9840457Z 
2025-09-07T08:14:18.9840544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9840736Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9840792Z     return mod(**inputs)
2025-09-07T08:14:18.9841028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9841092Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9841358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9841428Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9841638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9841738Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9841967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9842038Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9842267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9842338Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9842563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9842673Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9842676Z 
2025-09-07T08:14:18.9842765Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9842948Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9843003Z     return mod(**inputs)
2025-09-07T08:14:18.9843273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9843337Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9843601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9843665Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9843877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9843950Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9844182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9844258Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9844486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9844564Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9844798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9844914Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9844918Z 
2025-09-07T08:14:18.9845013Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9845194Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9845251Z     return mod(**inputs)
2025-09-07T08:14:18.9845477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9845539Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9845769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9845829Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9846047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9846113Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9846339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9846413Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9846639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9846716Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9846997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9847146Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9847152Z 
2025-09-07T08:14:18.9847240Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9847459Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9847521Z     return mod(**inputs)
2025-09-07T08:14:18.9847753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9847819Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9848048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9848111Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9848330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9848396Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9848629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9848731Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9848961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9849036Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9849294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9849369Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9849372Z 
2025-09-07T08:14:18.9849459Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9849645Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9849703Z     return mod(**inputs)
2025-09-07T08:14:18.9849934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9849999Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9850230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9850291Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9850505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9850573Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9850809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9850879Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9851116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9851189Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9851415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9851519Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9851523Z 
2025-09-07T08:14:18.9851611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9851799Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9851857Z     return mod(**inputs)
2025-09-07T08:14:18.9852093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9852156Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9852385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9852492Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9852706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9852781Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9853046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9853120Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9853353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9853424Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9853652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9853742Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9853747Z 
2025-09-07T08:14:18.9853836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9854021Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9854079Z     return mod(**inputs)
2025-09-07T08:14:18.9854352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9854421Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9854662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9854756Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9854971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9855048Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9855274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9855350Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9855576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9855650Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9855881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9855973Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9855976Z 
2025-09-07T08:14:18.9856068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9856248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9856306Z     return mod(**inputs)
2025-09-07T08:14:18.9856543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9856615Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9856847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9856906Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9857125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9857246Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9857476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9857555Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9857784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9857866Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9858094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9858267Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9858272Z 
2025-09-07T08:14:18.9858359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9858541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9858636Z     return mod(**inputs)
2025-09-07T08:14:18.9858865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9858933Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9859164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9859230Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9859450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9859520Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9859754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9859826Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9860087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9860170Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9860432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9860509Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9860512Z 
2025-09-07T08:14:18.9860586Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9860677Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9860862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9860922Z     return mod(**inputs)
2025-09-07T08:14:18.9861162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9861226Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9861462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9861525Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9861739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9861814Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9862039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9862129Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9862356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9862446Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9862676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9862745Z     return self.weight * hidden_states
2025-09-07T08:14:18.9862749Z 
2025-09-07T08:14:18.9862849Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9863031Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9863090Z     return mod(**inputs)
2025-09-07T08:14:18.9863327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9863389Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9863628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9863724Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9863936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9864001Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9864229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9864341Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9864570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9864678Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9864902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9864972Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9864977Z 
2025-09-07T08:14:18.9865067Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9865248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9865307Z     return mod(**inputs)
2025-09-07T08:14:18.9865537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9865637Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9865873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9865968Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9866188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9866254Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9866483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9866562Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9866786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9866894Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9867122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9867195Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9867199Z 
2025-09-07T08:14:18.9867287Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9867468Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9867526Z     return mod(**inputs)
2025-09-07T08:14:18.9867753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9867818Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9868048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9868110Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9868318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9868386Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9868622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9868701Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9868933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9869031Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9869259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9869367Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9869370Z 
2025-09-07T08:14:18.9869442Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9869540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9869724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9869820Z     return mod(**inputs)
2025-09-07T08:14:18.9870059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9870124Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9870361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9870423Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9870632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9870700Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9870927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9871007Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9871267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T08:14:18.9871376Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9871635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9871703Z     return self.weight * hidden_states
2025-09-07T08:14:18.9871706Z 
2025-09-07T08:14:18.9871797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9871979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9872040Z     return mod(**inputs)
2025-09-07T08:14:18.9872269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9872331Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9872567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9872630Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9872849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9872919Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9873150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9873219Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9873444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9873518Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9873743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9873813Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9873819Z 
2025-09-07T08:14:18.9873908Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9874089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9874152Z     return mod(**inputs)
2025-09-07T08:14:18.9874392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9874461Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9874690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9874790Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9875019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9875095Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9875335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9875446Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9875680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9875755Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9875979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9876049Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9876053Z 
2025-09-07T08:14:18.9876148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9876345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9876404Z     return mod(**inputs)
2025-09-07T08:14:18.9876637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9876712Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9876975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9877047Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9877312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9877384Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9877619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9877691Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9877926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9877996Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9878227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9878353Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9878356Z 
2025-09-07T08:14:18.9878451Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9878645Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9878701Z     return mod(**inputs)
2025-09-07T08:14:18.9878940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9879006Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9879237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9879305Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9879518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9879592Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9879817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9879889Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9880117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9880188Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9880415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9880559Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9880562Z 
2025-09-07T08:14:18.9880657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9880839Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9880894Z     return mod(**inputs)
2025-09-07T08:14:18.9881168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9881233Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9881467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9881529Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9881742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9881816Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9882047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9882118Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9882344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9882454Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9882688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9882837Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9882840Z 
2025-09-07T08:14:18.9882938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9883125Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9883190Z     return mod(**inputs)
2025-09-07T08:14:18.9883420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9883483Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9883715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9883780Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9883998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9884064Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9888422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9888533Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9888804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9888894Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9889139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9889295Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9889304Z 
2025-09-07T08:14:18.9889409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9889618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9889682Z     return mod(**inputs)
2025-09-07T08:14:18.9889934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9890010Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9890242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9890309Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9890600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9890677Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9890919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9890994Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9891264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9891345Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9891572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9891653Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9891657Z 
2025-09-07T08:14:18.9891753Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9891954Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9892020Z     return mod(**inputs)
2025-09-07T08:14:18.9892258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9892328Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9892591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9892660Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9892912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9892985Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9893218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9893289Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9893517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9893590Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9893817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9893927Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9893934Z 
2025-09-07T08:14:18.9894031Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9894226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9894288Z     return mod(**inputs)
2025-09-07T08:14:18.9894520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9894590Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9894819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9894891Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9895104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9895175Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9895406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9895478Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9895713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9895786Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9896015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9896108Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9896144Z 
2025-09-07T08:14:18.9896236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9896426Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9896482Z     return mod(**inputs)
2025-09-07T08:14:18.9896718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9896812Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9897046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9897112Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9897408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9897485Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9897724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9897801Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9898032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9898105Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9898377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9898476Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9898480Z 
2025-09-07T08:14:18.9898608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9898805Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9898868Z     return mod(**inputs)
2025-09-07T08:14:18.9899112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9899181Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9899415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9899476Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9899692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9899761Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9899988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9900067Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9900295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9900370Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9900597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9900692Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9900696Z 
2025-09-07T08:14:18.9900790Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9900976Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9901042Z     return mod(**inputs)
2025-09-07T08:14:18.9901275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9901336Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9901573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9901636Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9901849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9901951Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9902182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T08:14:18.9902253Z     self_attention_outputs = self.layer[0](
2025-09-07T08:14:18.9902481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T08:14:18.9902591Z     attention_output = self.SelfAttention(
2025-09-07T08:14:18.9902818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9902894Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9902898Z 
2025-09-07T08:14:18.9902973Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9903231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9903429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9903490Z     return mod(**inputs)
2025-09-07T08:14:18.9903720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9903786Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9904020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9904173Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9904392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9904510Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9904744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9904819Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9905044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T08:14:18.9905142Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9905373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9905441Z     return self.weight * hidden_states
2025-09-07T08:14:18.9905446Z 
2025-09-07T08:14:18.9905543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9905726Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9905784Z     return mod(**inputs)
2025-09-07T08:14:18.9906023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9906086Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9906318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9906382Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9906593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9906660Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9906889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9906966Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9907190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9907268Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9907495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T08:14:18.9907564Z     query_states = self.q(hidden_states)
2025-09-07T08:14:18.9907567Z 
2025-09-07T08:14:18.9907660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9907895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9907957Z     return mod(**inputs)
2025-09-07T08:14:18.9908191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9908299Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9908542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9908604Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9908821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9908888Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9909119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9909189Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9909415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9909492Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9909718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T08:14:18.9909826Z     key_states = self.k(current_states)
2025-09-07T08:14:18.9909829Z 
2025-09-07T08:14:18.9909917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9910133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9910198Z     return mod(**inputs)
2025-09-07T08:14:18.9910428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9910492Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9910722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9910781Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9910994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9911058Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9911288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9911357Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9911584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9911655Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9911878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9912004Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9912007Z 
2025-09-07T08:14:18.9912094Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9912278Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9912332Z     return mod(**inputs)
2025-09-07T08:14:18.9912563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9912628Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9912856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9912918Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9913127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9913192Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9913466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9913540Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9913767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9913887Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9914117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9914230Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9914233Z 
2025-09-07T08:14:18.9914321Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9914506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9914560Z     return mod(**inputs)
2025-09-07T08:14:18.9914789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9914852Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9915079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9915141Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9915384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9915454Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9915711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9915785Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9916012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9916086Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9916317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T08:14:18.9916425Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T08:14:18.9916428Z 
2025-09-07T08:14:18.9916515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9916698Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9916753Z     return mod(**inputs)
2025-09-07T08:14:18.9916986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9917048Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9917282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9917341Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9917552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9917626Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9917853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9917924Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9918151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9918223Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9918447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T08:14:18.9918595Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T08:14:18.9918599Z 
2025-09-07T08:14:18.9918690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9918872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9918967Z     return mod(**inputs)
2025-09-07T08:14:18.9919205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9919268Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9919542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9919607Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9919821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9919887Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9920114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9920184Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9920409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9920481Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9920706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T08:14:18.9920810Z     value_states = self.v(current_states)
2025-09-07T08:14:18.9920814Z 
2025-09-07T08:14:18.9920902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9921084Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9921174Z     return mod(**inputs)
2025-09-07T08:14:18.9921406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9921470Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9921701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9921765Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9921978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9922043Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9922275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9922343Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9922571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9922642Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9922866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9922967Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9922972Z 
2025-09-07T08:14:18.9923061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9923250Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9923307Z     return mod(**inputs)
2025-09-07T08:14:18.9923541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9923612Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9923841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9923911Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9924124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9924189Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9924425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9924528Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9924762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9924834Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9925122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9925222Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9925226Z 
2025-09-07T08:14:18.9925318Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9925514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9925573Z     return mod(**inputs)
2025-09-07T08:14:18.9925816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9925884Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9926117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9926185Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9926434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9926513Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9926742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9926848Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9927084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9927157Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9927384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T08:14:18.9927476Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T08:14:18.9927480Z 
2025-09-07T08:14:18.9927570Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9927756Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9927816Z     return mod(**inputs)
2025-09-07T08:14:18.9928047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9928110Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9928339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9928399Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9928610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9928677Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9928902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9928973Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9929197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9929272Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9929499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T08:14:18.9929591Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:14:18.9929594Z 
2025-09-07T08:14:18.9929683Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9929867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9929961Z     return mod(**inputs)
2025-09-07T08:14:18.9930191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9930252Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9930482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9930579Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9930795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9930862Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9931091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9931164Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9931391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T08:14:18.9931467Z     attention_output = self.EncDecAttention(
2025-09-07T08:14:18.9931692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T08:14:18.9931764Z     attn_output = self.o(attn_output)
2025-09-07T08:14:18.9931767Z 
2025-09-07T08:14:18.9931888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9932070Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9932131Z     return mod(**inputs)
2025-09-07T08:14:18.9932394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9932460Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9932687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9932747Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9932964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9933029Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9933259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T08:14:18.9933332Z     cross_attention_outputs = self.layer[1](
2025-09-07T08:14:18.9933557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T08:14:18.9933684Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T08:14:18.9933687Z 
2025-09-07T08:14:18.9933760Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9933851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9934031Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9934093Z     return mod(**inputs)
2025-09-07T08:14:18.9934323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9934384Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9934619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9934686Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9934900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9934967Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9935192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9935278Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9935505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T08:14:18.9935626Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T08:14:18.9935852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T08:14:18.9935928Z     return self.weight * hidden_states
2025-09-07T08:14:18.9935931Z 
2025-09-07T08:14:18.9936062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9936243Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9936308Z     return mod(**inputs)
2025-09-07T08:14:18.9936541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9936611Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9936840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9936901Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9937123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9937235Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9937466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9937581Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9937820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9937970Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9938204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T08:14:18.9938287Z     hidden_states = self.wi(hidden_states)
2025-09-07T08:14:18.9938290Z 
2025-09-07T08:14:18.9938388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9938585Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9938643Z     return mod(**inputs)
2025-09-07T08:14:18.9938878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9938955Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9939186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9939258Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9939477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9939548Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9939785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9939866Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9940097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9940203Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9940432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T08:14:18.9940514Z     hidden_states = self.act(hidden_states)
2025-09-07T08:14:18.9940517Z 
2025-09-07T08:14:18.9940608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9940802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9940857Z     return mod(**inputs)
2025-09-07T08:14:18.9941094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T08:14:18.9941157Z     decoder_outputs = self.decoder(
2025-09-07T08:14:18.9941437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T08:14:18.9941499Z     layer_outputs = layer_module(
2025-09-07T08:14:18.9941712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:14:18.9941816Z     return super().__call__(*args, **kwargs)
2025-09-07T08:14:18.9942046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T08:14:18.9942123Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T08:14:18.9942347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T08:14:18.9942446Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T08:14:18.9942675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T08:14:18.9942746Z     hidden_states = self.wo(hidden_states)
2025-09-07T08:14:18.9942749Z 
2025-09-07T08:14:18.9942822Z cudagraph partition due to non gpu ops
2025-09-07T08:14:18.9942910Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9943093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9943184Z     return mod(**inputs)
2025-09-07T08:14:18.9943415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward
2025-09-07T08:14:18.9943558Z     sequence_output = sequence_output * (self.model_dim**-0.5)
2025-09-07T08:14:18.9943561Z 
2025-09-07T08:14:18.9943649Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:14:18.9943835Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:14:18.9943890Z     return mod(**inputs)
2025-09-07T08:14:18.9944119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward
2025-09-07T08:14:18.9944197Z     lm_logits = self.lm_head(sequence_output)
2025-09-07T08:14:18.9944200Z 
2025-09-07T08:15:17.2550936Z pass
2025-09-07T08:15:17.2551349Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:15:24.2042603Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:15:24.2043487Z   import pynvml  # type: ignore[import]
2025-09-07T08:15:26.2187614Z 
2025-09-07T08:15:38.5646677Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:15:38.5647018Z loading model: 0it [00:12, ?it/s]
2025-09-07T08:15:38.6514908Z cpu  eval  hf_T5_large                        
2025-09-07T08:15:39.0497228Z pass_due_to_skip
2025-09-07T08:15:39.0500196Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:15:41.3696871Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:15:41.3697826Z   import pynvml  # type: ignore[import]
2025-09-07T08:15:43.3787852Z 
2025-09-07T08:15:49.6300198Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:15:49.6300541Z loading model: 0it [00:06, ?it/s]
2025-09-07T08:15:49.6687429Z cpu  eval  hf_distil_whisper                  
2025-09-07T08:15:56.9696140Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:15:59.0598346Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:01.1537164Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:09.7600354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7600822Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7601192Z     return mod(*inputs)
2025-09-07T08:16:09.7602033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7602471Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7602895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 677, in forward
2025-09-07T08:16:09.7603488Z     inputs_embeds = nn.functional.gelu(self.conv1(input_features))
2025-09-07T08:16:09.7603679Z 
2025-09-07T08:16:09.7603797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7604168Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7604493Z     return mod(*inputs)
2025-09-07T08:16:09.7604879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7605285Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7605799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 677, in forward
2025-09-07T08:16:09.7606257Z     inputs_embeds = nn.functional.gelu(self.conv1(input_features))
2025-09-07T08:16:09.7606432Z 
2025-09-07T08:16:09.7606617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7606982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7607309Z     return mod(*inputs)
2025-09-07T08:16:09.7607696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7608106Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7608501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 678, in forward
2025-09-07T08:16:09.7608950Z     inputs_embeds = nn.functional.gelu(self.conv2(inputs_embeds))
2025-09-07T08:16:09.7609136Z 
2025-09-07T08:16:09.7609245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7609600Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7609921Z     return mod(*inputs)
2025-09-07T08:16:09.7610278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7610676Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7611060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7611458Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7611809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7612190Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7612595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 409, in forward
2025-09-07T08:16:09.7613034Z     hidden_states = self.self_attn_layer_norm(hidden_states)
2025-09-07T08:16:09.7613199Z 
2025-09-07T08:16:09.7613396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7613758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7614096Z     return mod(*inputs)
2025-09-07T08:16:09.7614466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7615787Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7616182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7616580Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7616925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7617349Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7617821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 409, in forward
2025-09-07T08:16:09.7618266Z     hidden_states = self.self_attn_layer_norm(hidden_states)
2025-09-07T08:16:09.7618433Z 
2025-09-07T08:16:09.7618536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7618891Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7619217Z     return mod(*inputs)
2025-09-07T08:16:09.7619598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7620010Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7620395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7620838Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7621188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7621556Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7621982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7622405Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7622830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7623269Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7623431Z 
2025-09-07T08:16:09.7623537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7623883Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7624210Z     return mod(*inputs)
2025-09-07T08:16:09.7624579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7624981Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7625367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7625756Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7626098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7626458Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7626856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7627273Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7627686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7628180Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7628402Z 
2025-09-07T08:16:09.7628499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7628873Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7629188Z     return mod(*inputs)
2025-09-07T08:16:09.7629546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7629991Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7630375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7630771Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7631101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7631503Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7631894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7632306Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7632708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7633191Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7633423Z 
2025-09-07T08:16:09.7633499Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7633706Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7633906Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7634092Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7634312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7634694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7635026Z     return mod(*inputs)
2025-09-07T08:16:09.7635438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7635844Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7636234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7636628Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7636976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7637329Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7637709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7638122Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7638529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7638946Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7639389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7639877Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7640065Z 
2025-09-07T08:16:09.7640159Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7640516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7640833Z     return mod(*inputs)
2025-09-07T08:16:09.7641188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7641608Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7641989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7642381Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7642722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7643068Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7643470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7643925Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7644339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7644760Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7645200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7645706Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7645878Z 
2025-09-07T08:16:09.7645979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7646337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7646650Z     return mod(*inputs)
2025-09-07T08:16:09.7647037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7647439Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7647834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7648237Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7648664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7649041Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7649521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7649971Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7650407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7650808Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7650951Z 
2025-09-07T08:16:09.7651063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7651425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7651753Z     return mod(*inputs)
2025-09-07T08:16:09.7652132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7652539Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7652949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7653349Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7653723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7654088Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7654493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 420, in forward
2025-09-07T08:16:09.7654928Z     hidden_states = self.final_layer_norm(hidden_states)
2025-09-07T08:16:09.7655092Z 
2025-09-07T08:16:09.7655205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7655571Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7655889Z     return mod(*inputs)
2025-09-07T08:16:09.7656264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7656665Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7657052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7657451Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7657857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7658292Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7658695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7659137Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7659305Z 
2025-09-07T08:16:09.7659452Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7659805Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7660126Z     return mod(*inputs)
2025-09-07T08:16:09.7660496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7660893Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7661273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7661666Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7662002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7662364Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7662755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7663215Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7663598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7663971Z     return self.act(input)
2025-09-07T08:16:09.7664080Z 
2025-09-07T08:16:09.7664185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7664529Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7664842Z     return mod(*inputs)
2025-09-07T08:16:09.7665217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7665619Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7665998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7666392Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7666736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7667089Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7667479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7667893Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7668028Z 
2025-09-07T08:16:09.7668127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7668476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7668802Z     return mod(*inputs)
2025-09-07T08:16:09.7669164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7669567Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7669950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7670342Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7670685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7671052Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7671441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 409, in forward
2025-09-07T08:16:09.7671874Z     hidden_states = self.self_attn_layer_norm(hidden_states)
2025-09-07T08:16:09.7672088Z 
2025-09-07T08:16:09.7672182Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7672524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7672843Z     return mod(*inputs)
2025-09-07T08:16:09.7673201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7673637Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7674022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7674415Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7674754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7675103Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7675496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7675913Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7676332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7676802Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7676969Z 
2025-09-07T08:16:09.7677063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7677460Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7677785Z     return mod(*inputs)
2025-09-07T08:16:09.7678169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7678562Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7678959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7679360Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7679714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7680085Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7680488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7680902Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7681317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7681807Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7682024Z 
2025-09-07T08:16:09.7682130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7682482Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7682799Z     return mod(*inputs)
2025-09-07T08:16:09.7683170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7683573Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7683954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7684350Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7684697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7685065Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7685457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7685862Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7686308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7686793Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7687013Z 
2025-09-07T08:16:09.7687133Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7687339Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7687526Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7687721Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7687945Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7688298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7688608Z     return mod(*inputs)
2025-09-07T08:16:09.7688971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7689373Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7689753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7690145Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7690511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7690874Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7691301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7691713Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7692109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7692528Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7692980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7693463Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7693643Z 
2025-09-07T08:16:09.7693748Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7694088Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7694404Z     return mod(*inputs)
2025-09-07T08:16:09.7694767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7695161Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7695538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7695921Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7696260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7696620Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7697010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7697406Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7697864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7698278Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7698743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7699203Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7699362Z 
2025-09-07T08:16:09.7699455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7699844Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7700155Z     return mod(*inputs)
2025-09-07T08:16:09.7700515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7700915Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7701327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7701719Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7702054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7702406Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7702793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7703356Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7703769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7704173Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7704302Z 
2025-09-07T08:16:09.7704401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7704826Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7705150Z     return mod(*inputs)
2025-09-07T08:16:09.7705582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7706007Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7706407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7706797Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7707143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7707503Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7707896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.7708304Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.7708432Z 
2025-09-07T08:16:09.7708531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7708882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7709197Z     return mod(*inputs)
2025-09-07T08:16:09.7709557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7709947Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7710331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7710725Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7711064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7711412Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7711795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7712235Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7712407Z 
2025-09-07T08:16:09.7712503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7712843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7713154Z     return mod(*inputs)
2025-09-07T08:16:09.7713509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7713963Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7714345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7714731Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7715114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7715469Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7715860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7716302Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7716684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7717015Z     return self.act(input)
2025-09-07T08:16:09.7717128Z 
2025-09-07T08:16:09.7717223Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7717564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7717878Z     return mod(*inputs)
2025-09-07T08:16:09.7718270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7718675Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7719059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7719483Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7719821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7720170Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7720563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7720965Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7721096Z 
2025-09-07T08:16:09.7721195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7721532Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7721843Z     return mod(*inputs)
2025-09-07T08:16:09.7722199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7722597Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7722977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7723364Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7723702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7724053Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7724441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7724848Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7725248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7725684Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7725856Z 
2025-09-07T08:16:09.7725949Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7726289Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7726601Z     return mod(*inputs)
2025-09-07T08:16:09.7726951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7727392Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7727772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7728164Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7728497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7728890Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7729286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7729698Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7730101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7730581Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7730827Z 
2025-09-07T08:16:09.7730923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7731261Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7731579Z     return mod(*inputs)
2025-09-07T08:16:09.7743485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7743994Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7744477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7744940Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7745306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7745680Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7746094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7746525Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7746946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7747446Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7747689Z 
2025-09-07T08:16:09.7747771Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7747978Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7748176Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7748368Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7748596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7748963Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7749297Z     return mod(*inputs)
2025-09-07T08:16:09.7749699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7750107Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7750527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7750939Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7751291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7751660Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7752066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7752490Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7752907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7753379Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7753829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7754338Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7754565Z 
2025-09-07T08:16:09.7754671Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7755036Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7755363Z     return mod(*inputs)
2025-09-07T08:16:09.7755740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7756154Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7756542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7756937Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7757283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7757640Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7758070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7758491Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7758932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7759357Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7759794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7760259Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7760427Z 
2025-09-07T08:16:09.7760525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7760879Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7761190Z     return mod(*inputs)
2025-09-07T08:16:09.7761560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7761960Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7762348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7762743Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7763082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7763436Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7763829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7764240Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7764641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7765037Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7765176Z 
2025-09-07T08:16:09.7765271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7765618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7765936Z     return mod(*inputs)
2025-09-07T08:16:09.7766294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7766692Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7767073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7767514Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7767853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7768203Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7768631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7769072Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7769241Z 
2025-09-07T08:16:09.7769344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7769688Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7769996Z     return mod(*inputs)
2025-09-07T08:16:09.7770355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7770752Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7771131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7771520Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7771888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7772253Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7772682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7773123Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7773500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7773841Z     return self.act(input)
2025-09-07T08:16:09.7773955Z 
2025-09-07T08:16:09.7774053Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7774395Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7774708Z     return mod(*inputs)
2025-09-07T08:16:09.7775063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7775461Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7775850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7776250Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7776577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7776928Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7777319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7777793Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7777928Z 
2025-09-07T08:16:09.7778028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7778368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7778683Z     return mod(*inputs)
2025-09-07T08:16:09.7779040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7779438Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7779815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7780203Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7780536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7780953Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7781347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7781755Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7782165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7782636Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7782799Z 
2025-09-07T08:16:09.7782899Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7783246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7783554Z     return mod(*inputs)
2025-09-07T08:16:09.7783914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7784307Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7784690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7785078Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7785402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7785788Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7786179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7786641Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7787045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7787529Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7787756Z 
2025-09-07T08:16:09.7787851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7788201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7788514Z     return mod(*inputs)
2025-09-07T08:16:09.7788867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7789267Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7789652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7790046Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7790381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7790732Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7791119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7791526Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7791933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7792420Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7792644Z 
2025-09-07T08:16:09.7792721Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7792926Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7793124Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7793319Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7793530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7793870Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7794182Z     return mod(*inputs)
2025-09-07T08:16:09.7794543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7794999Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7795382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7795775Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7796156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7796503Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7796884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7797293Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7797685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7798111Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7798548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7799020Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7799203Z 
2025-09-07T08:16:09.7799335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7799678Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7799978Z     return mod(*inputs)
2025-09-07T08:16:09.7800956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7801382Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7801785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7802182Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7802510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7802858Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7803444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7803875Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7804297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7804722Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7805174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7805631Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7805789Z 
2025-09-07T08:16:09.7805889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7806245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7806560Z     return mod(*inputs)
2025-09-07T08:16:09.7806932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7807327Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7807701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7808085Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7808427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7808772Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7809159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7809650Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7810043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7810441Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7810621Z 
2025-09-07T08:16:09.7810716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7811056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7811379Z     return mod(*inputs)
2025-09-07T08:16:09.7811738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7812130Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7812508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7812898Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7813229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7813588Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7814041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.7814468Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.7814601Z 
2025-09-07T08:16:09.7814706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7815100Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7815423Z     return mod(*inputs)
2025-09-07T08:16:09.7815790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7816192Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7816571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7816961Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7817299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7817696Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7818084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7818519Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7818687Z 
2025-09-07T08:16:09.7818781Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7819118Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7819425Z     return mod(*inputs)
2025-09-07T08:16:09.7819781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7820165Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7820540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7820929Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7821269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7821619Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7822013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7822448Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7822826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7823206Z     return self.act(input)
2025-09-07T08:16:09.7823310Z 
2025-09-07T08:16:09.7823404Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7823745Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7824055Z     return mod(*inputs)
2025-09-07T08:16:09.7824454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7824849Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7825233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7825619Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7825955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7826303Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7826687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7827080Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7827213Z 
2025-09-07T08:16:09.7827310Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7827684Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7827999Z     return mod(*inputs)
2025-09-07T08:16:09.7828381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7828775Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7829151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7829535Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7829863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7830209Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7830609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7831020Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7831427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7831852Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7832015Z 
2025-09-07T08:16:09.7832113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7832455Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7832767Z     return mod(*inputs)
2025-09-07T08:16:09.7833125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7833517Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7833895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7834279Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7834616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7834964Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7835347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7835749Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7836146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7836666Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7836881Z 
2025-09-07T08:16:09.7837202Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7837541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7837849Z     return mod(*inputs)
2025-09-07T08:16:09.7838244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7838632Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7839007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7839390Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7839813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7840162Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7840550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7840950Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7841454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7841938Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7842159Z 
2025-09-07T08:16:09.7842236Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7842489Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7842677Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7842864Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7843075Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7843414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7843724Z     return mod(*inputs)
2025-09-07T08:16:09.7844079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7844469Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7844845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7845232Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7845566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7845917Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7846306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7846702Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7847095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7847510Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7847954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7848435Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7848614Z 
2025-09-07T08:16:09.7848712Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7849053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7849366Z     return mod(*inputs)
2025-09-07T08:16:09.7849718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7850113Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7850482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7850924Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7851258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7851609Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7852039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7852441Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7852848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7853258Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7853696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7854152Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7854309Z 
2025-09-07T08:16:09.7854404Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7854745Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7855059Z     return mod(*inputs)
2025-09-07T08:16:09.7855455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7855856Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7856265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7856655Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7856986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7857336Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7857762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7858168Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7858567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7858965Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7859090Z 
2025-09-07T08:16:09.7859187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7859520Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7859828Z     return mod(*inputs)
2025-09-07T08:16:09.7860181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7860573Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7860949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7861329Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7861656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7862004Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7862382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7862812Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7862980Z 
2025-09-07T08:16:09.7863071Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7863411Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7863717Z     return mod(*inputs)
2025-09-07T08:16:09.7864123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7864506Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7864879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7865305Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7865638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7865983Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7866365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7866817Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7867189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7867532Z     return self.act(input)
2025-09-07T08:16:09.7867634Z 
2025-09-07T08:16:09.7867729Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7868059Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7868366Z     return mod(*inputs)
2025-09-07T08:16:09.7868755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7869145Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7869551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7869940Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7870274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7870630Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7871011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7871402Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7871534Z 
2025-09-07T08:16:09.7871626Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7871966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7872277Z     return mod(*inputs)
2025-09-07T08:16:09.7872623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7873022Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7873393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7873783Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7874106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7874451Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7874843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7875267Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7875669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7876090Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7876254Z 
2025-09-07T08:16:09.7876345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7876688Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7876992Z     return mod(*inputs)
2025-09-07T08:16:09.7877338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7877776Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7878145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7878558Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7878998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7879341Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7879719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7880139Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7880532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7881022Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7881239Z 
2025-09-07T08:16:09.7881332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7881673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7881983Z     return mod(*inputs)
2025-09-07T08:16:09.7882387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7882788Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7883207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7883600Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7883952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7884312Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7884720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7885123Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7885522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7886015Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7886235Z 
2025-09-07T08:16:09.7886313Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7886512Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7886710Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7886902Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7887123Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7887467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7887799Z     return mod(*inputs)
2025-09-07T08:16:09.7888159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7888564Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7888951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7889339Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7889667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7890019Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7890427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7890829Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7891223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7891681Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7892124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7892652Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7892834Z 
2025-09-07T08:16:09.7892930Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7893285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7893605Z     return mod(*inputs)
2025-09-07T08:16:09.7893964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7894362Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7894743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7895126Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7895455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7895800Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7896246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7896670Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7897095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7897513Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7897985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7898450Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7898607Z 
2025-09-07T08:16:09.7898701Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7899038Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7899348Z     return mod(*inputs)
2025-09-07T08:16:09.7899705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7900100Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7900470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7900860Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7901193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7901547Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7901937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7902332Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7902731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7903270Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7903395Z 
2025-09-07T08:16:09.7903487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7903823Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7904137Z     return mod(*inputs)
2025-09-07T08:16:09.7904489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7904889Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7905367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7905750Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7906077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7906489Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7906879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.7907283Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.7907411Z 
2025-09-07T08:16:09.7907504Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7907852Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7908168Z     return mod(*inputs)
2025-09-07T08:16:09.7908526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7908918Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7909301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7909686Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7910079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7910430Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7910851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7911290Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7911461Z 
2025-09-07T08:16:09.7911554Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7911890Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7912205Z     return mod(*inputs)
2025-09-07T08:16:09.7912556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7912951Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7913323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7913709Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7914034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7914386Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7914767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7915198Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7915585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7915921Z     return self.act(input)
2025-09-07T08:16:09.7916033Z 
2025-09-07T08:16:09.7916125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7916473Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7916787Z     return mod(*inputs)
2025-09-07T08:16:09.7917160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7917550Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7917924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7918313Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7918660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7919058Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7919455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7919855Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7920027Z 
2025-09-07T08:16:09.7920135Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7920480Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7920784Z     return mod(*inputs)
2025-09-07T08:16:09.7921149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7921544Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7921926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7922316Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7922641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7923006Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7923429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7923853Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7924283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7924720Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7924882Z 
2025-09-07T08:16:09.7924972Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7925307Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7925623Z     return mod(*inputs)
2025-09-07T08:16:09.7925978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7926380Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7926771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7927166Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7927502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7927849Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7928233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7928636Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7929042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7929518Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7929736Z 
2025-09-07T08:16:09.7929834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7930192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7930524Z     return mod(*inputs)
2025-09-07T08:16:09.7930895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7931294Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7931686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7932082Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7932427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7932833Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7933228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7933651Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7934099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7934588Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7934814Z 
2025-09-07T08:16:09.7934902Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7935103Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7935300Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7935495Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7935718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7936067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7936387Z     return mod(*inputs)
2025-09-07T08:16:09.7936753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7937157Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7937648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7938048Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7938426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7938796Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7939192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7939601Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7940010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7940431Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7940889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7941374Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7941556Z 
2025-09-07T08:16:09.7941654Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7942004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7942324Z     return mod(*inputs)
2025-09-07T08:16:09.7942689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7943090Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7943462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7943857Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7944204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7944562Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7944949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7945360Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7945762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7946179Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7946622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7947117Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7947282Z 
2025-09-07T08:16:09.7947378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7947729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7948101Z     return mod(*inputs)
2025-09-07T08:16:09.7948464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7948857Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7949240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7949644Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7950016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7950381Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7950789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7951202Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7951654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7952056Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7952184Z 
2025-09-07T08:16:09.7952314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7952669Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7952992Z     return mod(*inputs)
2025-09-07T08:16:09.7953360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7953770Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7954148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7954541Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7954877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7955236Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7955618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7956052Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7956219Z 
2025-09-07T08:16:09.7956312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7956649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7956956Z     return mod(*inputs)
2025-09-07T08:16:09.7957308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7957700Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7958077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7958463Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7958792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7959133Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7959513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7959946Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7960330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.7960749Z     return self.act(input)
2025-09-07T08:16:09.7960857Z 
2025-09-07T08:16:09.7960951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7961296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7961658Z     return mod(*inputs)
2025-09-07T08:16:09.7962021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7962411Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7962795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7963186Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7963532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7963888Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7964275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.7964685Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.7964826Z 
2025-09-07T08:16:09.7964956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7965303Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7965611Z     return mod(*inputs)
2025-09-07T08:16:09.7966003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7966410Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7966797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7967195Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7967526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7967885Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7968273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7968689Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7969097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.7969527Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.7969696Z 
2025-09-07T08:16:09.7969789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7970132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7970446Z     return mod(*inputs)
2025-09-07T08:16:09.7970799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7971194Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7971574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7971971Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7972310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7972657Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7973047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7973455Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7973860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.7974393Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7974612Z 
2025-09-07T08:16:09.7974706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7975049Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7975403Z     return mod(*inputs)
2025-09-07T08:16:09.7975763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7976165Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7976540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7976936Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7977275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7977663Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7978051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7978460Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7978915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.7979414Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.7979637Z 
2025-09-07T08:16:09.7979754Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7979954Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7980149Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7980343Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.7980560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7980900Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7981212Z     return mod(*inputs)
2025-09-07T08:16:09.7981573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7981968Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7982355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7982743Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7983079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7983433Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7983823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7984229Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7984627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7985044Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7985498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.7985981Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.7986162Z 
2025-09-07T08:16:09.7986258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7986613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7986928Z     return mod(*inputs)
2025-09-07T08:16:09.7987294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7987692Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7988120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7988520Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7988857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7989252Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7989640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7990040Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7990445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.7990858Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.7991299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.7991763Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.7991925Z 
2025-09-07T08:16:09.7992019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7992364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7992727Z     return mod(*inputs)
2025-09-07T08:16:09.7993087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7993511Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7993893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7994282Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7994501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7994585Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7994847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.7994932Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.7995192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.7995268Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.7995276Z 
2025-09-07T08:16:09.7995371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7995560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7995628Z     return mod(*inputs)
2025-09-07T08:16:09.7995894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7995967Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7996228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7996294Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7996518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7996588Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7996858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.7996930Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.7996933Z 
2025-09-07T08:16:09.7997025Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7997216Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7997272Z     return mod(*inputs)
2025-09-07T08:16:09.7997611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7997680Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7997942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7998045Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.7998264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.7998345Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.7998602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.7998716Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.7998719Z 
2025-09-07T08:16:09.7998810Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.7998998Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.7999065Z     return mod(*inputs)
2025-09-07T08:16:09.7999329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.7999435Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.7999699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.7999772Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8000022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8000096Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8000365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8000475Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8000694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8000761Z     return self.act(input)
2025-09-07T08:16:09.8000764Z 
2025-09-07T08:16:09.8000857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8001056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8001116Z     return mod(*inputs)
2025-09-07T08:16:09.8001388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8001453Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8001712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8001784Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8002003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8002079Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8002341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8002427Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8002430Z 
2025-09-07T08:16:09.8002523Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8002710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8002776Z     return mod(*inputs)
2025-09-07T08:16:09.8003172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8003247Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8003518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8003681Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8003906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8003977Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8004300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8004389Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8004662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8004768Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8004772Z 
2025-09-07T08:16:09.8004868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8005068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8005130Z     return mod(*inputs)
2025-09-07T08:16:09.8005400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8005466Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8005795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8005870Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8006141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8006223Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8006484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8006568Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8006840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8006996Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8007000Z 
2025-09-07T08:16:09.8007099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8007289Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8007354Z     return mod(*inputs)
2025-09-07T08:16:09.8007624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8007691Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8007958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8008024Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8008250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8008321Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8008579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8008668Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8008926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8009095Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8009098Z 
2025-09-07T08:16:09.8009173Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8009250Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8009321Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8009392Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8009540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8009728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8009794Z     return mod(*inputs)
2025-09-07T08:16:09.8010064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8010166Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8010434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8010502Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8018714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8018838Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8019154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8019254Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8019549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8019653Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8020050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8020188Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8020229Z 
2025-09-07T08:16:09.8020339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8020556Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8020621Z     return mod(*inputs)
2025-09-07T08:16:09.8020913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8020993Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8021277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8021348Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8021575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8021655Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8021926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8022017Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8022287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8022385Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8022687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8022790Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8022794Z 
2025-09-07T08:16:09.8022899Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8023107Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8023168Z     return mod(*inputs)
2025-09-07T08:16:09.8023447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8023518Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8023781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8023847Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8024111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8024191Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8024454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8024583Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8024844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8024921Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8024925Z 
2025-09-07T08:16:09.8025025Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8025219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8025279Z     return mod(*inputs)
2025-09-07T08:16:09.8025550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8025617Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8025877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8025977Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8026200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8026269Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8026580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8026699Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8026703Z 
2025-09-07T08:16:09.8026796Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8026995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8027054Z     return mod(*inputs)
2025-09-07T08:16:09.8027325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8027389Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8027652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8027722Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8027946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8028019Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8028279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8028387Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8028602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8028668Z     return self.act(input)
2025-09-07T08:16:09.8028672Z 
2025-09-07T08:16:09.8028771Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8028967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8029032Z     return mod(*inputs)
2025-09-07T08:16:09.8029311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8029377Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8029647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8029712Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8029974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8030045Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8030314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8030426Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8030431Z 
2025-09-07T08:16:09.8030528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8030724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8030785Z     return mod(*inputs)
2025-09-07T08:16:09.8031060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8031125Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8031385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8031458Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8031673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8031747Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8032045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8032134Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8032475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8032584Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8032588Z 
2025-09-07T08:16:09.8032688Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8032875Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8032942Z     return mod(*inputs)
2025-09-07T08:16:09.8033208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8033274Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8033541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8033609Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8033830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8033898Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8034158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8034245Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8034506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8034671Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8034674Z 
2025-09-07T08:16:09.8034766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8034965Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8035024Z     return mod(*inputs)
2025-09-07T08:16:09.8035290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8035361Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8035621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8035690Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8035945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8036014Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8036281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8036394Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8036658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8036820Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8036824Z 
2025-09-07T08:16:09.8036901Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8036970Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8037038Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8037111Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8037202Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8037393Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8037450Z     return mod(*inputs)
2025-09-07T08:16:09.8037710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8037810Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8038070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8038170Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8038385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8038456Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8038715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8038797Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8039063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8039155Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8039456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8039582Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8039586Z 
2025-09-07T08:16:09.8039676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8039866Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8039922Z     return mod(*inputs)
2025-09-07T08:16:09.8040189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8040254Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8040513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8040578Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8040793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8040866Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8041123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8041202Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8041462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8041544Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8041874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8041973Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8041976Z 
2025-09-07T08:16:09.8042069Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8042288Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8042347Z     return mod(*inputs)
2025-09-07T08:16:09.8042614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8042677Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8042941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8043003Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8043222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8043292Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8043562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8043680Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8043943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8044052Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8044056Z 
2025-09-07T08:16:09.8044148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8044334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8044395Z     return mod(*inputs)
2025-09-07T08:16:09.8044660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8044734Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8044994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8045060Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8045277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8045347Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8045609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8045679Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8045683Z 
2025-09-07T08:16:09.8045778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8045962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8046021Z     return mod(*inputs)
2025-09-07T08:16:09.8046285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8046351Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8046617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8046682Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8046899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8046972Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8047236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8047349Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8047415Z 
2025-09-07T08:16:09.8047504Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8047689Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8047745Z     return mod(*inputs)
2025-09-07T08:16:09.8048007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8048109Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8048371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8048439Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8048649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8048717Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8048976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8049080Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8049294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8049356Z     return self.act(input)
2025-09-07T08:16:09.8049361Z 
2025-09-07T08:16:09.8049505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8049692Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8049782Z     return mod(*inputs)
2025-09-07T08:16:09.8050048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8050111Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8050376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8050440Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8050655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8050723Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8050981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8051057Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8051060Z 
2025-09-07T08:16:09.8051149Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8051332Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8051394Z     return mod(*inputs)
2025-09-07T08:16:09.8051654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8051721Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8051977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8052041Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8052250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8052319Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8052578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8052658Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8052916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8053017Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8053020Z 
2025-09-07T08:16:09.8053163Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8053351Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8053407Z     return mod(*inputs)
2025-09-07T08:16:09.8053670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8053775Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8054036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8054100Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8054312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8054385Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8054645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8054725Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8054982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8055133Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8055141Z 
2025-09-07T08:16:09.8055271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8055456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8055546Z     return mod(*inputs)
2025-09-07T08:16:09.8058293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8058384Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8058667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8058741Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8058967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8059042Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8059315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8059414Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8059683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8059858Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8059883Z 
2025-09-07T08:16:09.8059966Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8060042Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8060120Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8060188Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8060293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8060489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8060549Z     return mod(*inputs)
2025-09-07T08:16:09.8060829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8060898Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8061165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8061231Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8061454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8061536Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8061861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8061950Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8062211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8062346Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8062644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8062771Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8062775Z 
2025-09-07T08:16:09.8062881Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8063075Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8063141Z     return mod(*inputs)
2025-09-07T08:16:09.8063408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8063473Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8063741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8063841Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8064069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8064142Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8064494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8064580Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8064846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8064937Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8065229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8065332Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8065339Z 
2025-09-07T08:16:09.8065433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8065620Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8065677Z     return mod(*inputs)
2025-09-07T08:16:09.8065943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8066007Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8066264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8066331Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8066558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8066631Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8066891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8066971Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8067231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8067303Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8067308Z 
2025-09-07T08:16:09.8067403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8067588Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8067688Z     return mod(*inputs)
2025-09-07T08:16:09.8067951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8068013Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8068274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8068368Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8068583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8068650Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8068912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8069025Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8069031Z 
2025-09-07T08:16:09.8069121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8069306Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8069361Z     return mod(*inputs)
2025-09-07T08:16:09.8069657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8069724Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8069982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8070047Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8070306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8070379Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8070638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8070742Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8070955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8071021Z     return self.act(input)
2025-09-07T08:16:09.8071026Z 
2025-09-07T08:16:09.8071121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8071309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8071368Z     return mod(*inputs)
2025-09-07T08:16:09.8071631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8071695Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8071953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8072016Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8072228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8072293Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8072553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8072633Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8072636Z 
2025-09-07T08:16:09.8072724Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8072912Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8072968Z     return mod(*inputs)
2025-09-07T08:16:09.8073233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8073302Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8073961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8074026Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8074248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8074354Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8074619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8074701Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8074968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8075071Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8075075Z 
2025-09-07T08:16:09.8075168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8075359Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8075415Z     return mod(*inputs)
2025-09-07T08:16:09.8075681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8075780Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8076046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8076109Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8076323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8076434Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8076695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8076784Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8077043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8077207Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8077212Z 
2025-09-07T08:16:09.8077305Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8077492Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8077550Z     return mod(*inputs)
2025-09-07T08:16:09.8077813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8077881Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8078138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8078202Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8078415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8078481Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8078741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8078818Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8079081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8079240Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8079244Z 
2025-09-07T08:16:09.8079319Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8079390Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8079460Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8079570Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8079661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8079849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8079915Z     return mod(*inputs)
2025-09-07T08:16:09.8080219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8080290Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8080549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8080615Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8080837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8080906Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8081164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8081239Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8081498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8081620Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8081917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8082046Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8082049Z 
2025-09-07T08:16:09.8082175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8082363Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8082421Z     return mod(*inputs)
2025-09-07T08:16:09.8082690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8082759Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8083017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8083084Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8083296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8083364Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8083621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8083698Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8083959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8084044Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8084339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8084439Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8084444Z 
2025-09-07T08:16:09.8084535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8084722Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8084777Z     return mod(*inputs)
2025-09-07T08:16:09.8085038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8085103Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8085361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8085459Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8085671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8085738Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8085998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8086111Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8086368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8086438Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8086442Z 
2025-09-07T08:16:09.8086533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8086714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8086773Z     return mod(*inputs)
2025-09-07T08:16:09.8087032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8087097Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8087396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8087463Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8087683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8087755Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8088055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8088129Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8088132Z 
2025-09-07T08:16:09.8088225Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8088417Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8088476Z     return mod(*inputs)
2025-09-07T08:16:09.8088743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8088813Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8089075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8089145Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8089359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8089435Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8089693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8089805Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8089808Z 
2025-09-07T08:16:09.8089898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8090082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8090145Z     return mod(*inputs)
2025-09-07T08:16:09.8090407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8090473Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8090733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8090799Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8091016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8091082Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8091389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8091492Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8091707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8091802Z     return self.act(input)
2025-09-07T08:16:09.8091805Z 
2025-09-07T08:16:09.8091897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8092086Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8092141Z     return mod(*inputs)
2025-09-07T08:16:09.8092411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8092473Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8092731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8092795Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8093010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8093086Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8093378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8093452Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8093459Z 
2025-09-07T08:16:09.8093552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8093786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8093848Z     return mod(*inputs)
2025-09-07T08:16:09.8094113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8094186Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8094446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8094508Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8094724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8094792Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8095053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8095133Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8095391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8095494Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8095500Z 
2025-09-07T08:16:09.8095589Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8095780Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8095837Z     return mod(*inputs)
2025-09-07T08:16:09.8096105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8096168Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8096428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8096496Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8096711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8096783Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8097082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8097162Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8097426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8097657Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8097661Z 
2025-09-07T08:16:09.8097758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8097943Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8098005Z     return mod(*inputs)
2025-09-07T08:16:09.8098268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8098330Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8098594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8098656Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8098869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8098938Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8099228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8099309Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8099569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8099762Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8099766Z 
2025-09-07T08:16:09.8099842Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8099920Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8099990Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8100059Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8100154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8100340Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8100402Z     return mod(*inputs)
2025-09-07T08:16:09.8100667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8100731Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8100993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8101054Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8101267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8101334Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8101590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8101667Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8101931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8102019Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8102313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8102438Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8102442Z 
2025-09-07T08:16:09.8102532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8102716Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8102810Z     return mod(*inputs)
2025-09-07T08:16:09.8103251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8103318Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8103638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8103703Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8103921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8103989Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8104253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8104332Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8104591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8104678Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8104968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8105124Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8105128Z 
2025-09-07T08:16:09.8105218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8105404Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8105459Z     return mod(*inputs)
2025-09-07T08:16:09.8105776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8105845Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8106106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8106178Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8106388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8106458Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8106724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8106802Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8107067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8107141Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8107145Z 
2025-09-07T08:16:09.8107238Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8107422Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8107480Z     return mod(*inputs)
2025-09-07T08:16:09.8107744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8107811Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8108074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8108136Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8108347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8108420Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8108680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8108860Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8108863Z 
2025-09-07T08:16:09.8108951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8109138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8109193Z     return mod(*inputs)
2025-09-07T08:16:09.8109493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8109560Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8109816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8109881Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8110096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8110162Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8110425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8110529Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8110741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8110844Z     return self.act(input)
2025-09-07T08:16:09.8110847Z 
2025-09-07T08:16:09.8110942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8111127Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8111182Z     return mod(*inputs)
2025-09-07T08:16:09.8111489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8111554Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8111816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8111882Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8112094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8112162Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8112422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8112496Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8112499Z 
2025-09-07T08:16:09.8112586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8112770Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8112828Z     return mod(*inputs)
2025-09-07T08:16:09.8113086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8113151Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8113414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8113481Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8113694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8113760Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8114021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8114098Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8114361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8114461Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8114504Z 
2025-09-07T08:16:09.8114594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8114782Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8114837Z     return mod(*inputs)
2025-09-07T08:16:09.8115103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8115199Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8115461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8115522Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8115735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8115806Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8116064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8116148Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8116405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8116605Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8116609Z 
2025-09-07T08:16:09.8116705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8116888Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8116948Z     return mod(*inputs)
2025-09-07T08:16:09.8117245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8117319Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8117579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8117645Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8117857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8117924Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8118189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8118266Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8118523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8118681Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8118685Z 
2025-09-07T08:16:09.8118757Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8118829Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8118896Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8118965Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8119057Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8119239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8119300Z     return mod(*inputs)
2025-09-07T08:16:09.8119558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8119622Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8119879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8119942Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8120157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8120264Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8120525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8120599Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8120858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8120989Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8121285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8121416Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8121421Z 
2025-09-07T08:16:09.8121513Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8121706Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8121768Z     return mod(*inputs)
2025-09-07T08:16:09.8122035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8122112Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8122406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8122482Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8122698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8122772Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8123074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8123157Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8123426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8123511Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8123803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8123911Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8123915Z 
2025-09-07T08:16:09.8124007Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8124196Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8124254Z     return mod(*inputs)
2025-09-07T08:16:09.8124521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8124586Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8124846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8124915Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8125126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8125202Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8125463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8125541Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8125807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8125882Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8125885Z 
2025-09-07T08:16:09.8125983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8126207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8126270Z     return mod(*inputs)
2025-09-07T08:16:09.8126534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8126599Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8126902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8126966Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8127186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8127255Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8127516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8127593Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8127598Z 
2025-09-07T08:16:09.8127690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8127881Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8127938Z     return mod(*inputs)
2025-09-07T08:16:09.8128237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8128304Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8128565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8128634Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8128884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8128960Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8129221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8129331Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8129335Z 
2025-09-07T08:16:09.8129434Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8129620Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8129684Z     return mod(*inputs)
2025-09-07T08:16:09.8129948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8130013Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8130280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8130345Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8130566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8130636Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8130901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8131004Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8131215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8131286Z     return self.act(input)
2025-09-07T08:16:09.8131290Z 
2025-09-07T08:16:09.8131378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8131570Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8131633Z     return mod(*inputs)
2025-09-07T08:16:09.8131899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8132002Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8132263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8132335Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8132551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8132657Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8132930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8133008Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8133012Z 
2025-09-07T08:16:09.8133119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8133311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8133378Z     return mod(*inputs)
2025-09-07T08:16:09.8133649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8133717Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8133986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8134086Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8134314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8134386Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8134647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8134771Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8135035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8135149Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8135153Z 
2025-09-07T08:16:09.8135254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8135456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8135521Z     return mod(*inputs)
2025-09-07T08:16:09.8135792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8135865Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8136126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8136198Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8136418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8136494Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8136760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8136843Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8137109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8137269Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8137273Z 
2025-09-07T08:16:09.8137376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8137611Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8137674Z     return mod(*inputs)
2025-09-07T08:16:09.8137946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8138047Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8138314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8138380Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8138600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8138725Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8138990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8139076Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8139338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8139506Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8139512Z 
2025-09-07T08:16:09.8139589Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8139659Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8139735Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8139803Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8139902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8140152Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8140213Z     return mod(*inputs)
2025-09-07T08:16:09.8140484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8140550Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8140855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8140922Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8141140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8141215Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8141475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8141563Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8141822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8141919Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8142219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8142344Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8142348Z 
2025-09-07T08:16:09.8142448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8142634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8142699Z     return mod(*inputs)
2025-09-07T08:16:09.8142962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8143031Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8143298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8143361Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8143583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8143655Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8143922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8144035Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8144295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8144390Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8144718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8144826Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8144830Z 
2025-09-07T08:16:09.8144924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8145113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8145179Z     return mod(*inputs)
2025-09-07T08:16:09.8145442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8145516Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8145777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8145849Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8146097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8146171Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8146437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8146517Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8146822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8146898Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8146903Z 
2025-09-07T08:16:09.8146995Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8147187Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8147244Z     return mod(*inputs)
2025-09-07T08:16:09.8147516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8147580Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8147843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8147906Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8148118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8148187Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8148442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8148559Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8148563Z 
2025-09-07T08:16:09.8148650Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8148831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8148893Z     return mod(*inputs)
2025-09-07T08:16:09.8149153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8149216Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8149478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8149540Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8149754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8149861Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8150122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8150223Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8150476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8150538Z     return self.act(input)
2025-09-07T08:16:09.8150541Z 
2025-09-07T08:16:09.8150630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8150815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8150871Z     return mod(*inputs)
2025-09-07T08:16:09.8151135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8151198Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8151454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8151517Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8151728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8151829Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8152093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8152167Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8152171Z 
2025-09-07T08:16:09.8152263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8152486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8152546Z     return mod(*inputs)
2025-09-07T08:16:09.8152810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8152874Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8153130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8153195Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8153407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8153473Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8153732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8153812Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8154070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8154175Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8154179Z 
2025-09-07T08:16:09.8154269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8154454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8154510Z     return mod(*inputs)
2025-09-07T08:16:09.8154776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8154838Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8155094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8155160Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8155371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8155475Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8155733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8155809Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8156071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8156256Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8156260Z 
2025-09-07T08:16:09.8156351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8156533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8156593Z     return mod(*inputs)
2025-09-07T08:16:09.8156854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8156919Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8157177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8157237Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8157450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8157550Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8157808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8157890Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8158193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8158352Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8158359Z 
2025-09-07T08:16:09.8158431Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8158500Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8158567Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8158633Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8158723Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8158908Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8158968Z     return mod(*inputs)
2025-09-07T08:16:09.8159229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8159290Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8159552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8159613Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8159827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8159893Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8160150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8160230Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8160486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8160573Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8160865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8160989Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8160992Z 
2025-09-07T08:16:09.8161083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8161320Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8161380Z     return mod(*inputs)
2025-09-07T08:16:09.8161645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8161752Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8162012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8162075Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8162290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8162357Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8162617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8162695Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8162955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8163037Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8163356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8163460Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8163464Z 
2025-09-07T08:16:09.8163552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8163739Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8163828Z     return mod(*inputs)
2025-09-07T08:16:09.8164092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8164163Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8164421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8164488Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8164702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8164771Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8165027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8165103Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8165363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8165435Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8165438Z 
2025-09-07T08:16:09.8165531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8165714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8165770Z     return mod(*inputs)
2025-09-07T08:16:09.8166035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8166101Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8166359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8166421Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8166633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8166703Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8166960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8167067Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8167069Z 
2025-09-07T08:16:09.8167158Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8167341Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8167430Z     return mod(*inputs)
2025-09-07T08:16:09.8167693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8167758Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8168025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8168092Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8168302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8168369Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8168633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8168738Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8168742Z 
2025-09-07T08:16:09.8168833Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8169047Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8169107Z     return mod(*inputs)
2025-09-07T08:16:09.8169369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8169432Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8169722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8169785Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8170018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8170087Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8170364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8170477Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8170690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8170755Z     return self.act(input)
2025-09-07T08:16:09.8170758Z 
2025-09-07T08:16:09.8170851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8171048Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8171107Z     return mod(*inputs)
2025-09-07T08:16:09.8171374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8171447Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8171707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8171778Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8171996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8172067Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8172336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8172413Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8172417Z 
2025-09-07T08:16:09.8172514Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8172703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8172798Z     return mod(*inputs)
2025-09-07T08:16:09.8173070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8173136Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8173443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8173509Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8173729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8173800Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8174059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8174149Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8174409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8174522Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8174526Z 
2025-09-07T08:16:09.8174618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8174840Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8174905Z     return mod(*inputs)
2025-09-07T08:16:09.8175168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8175239Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8175530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8175604Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8175819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8175892Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8176163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8176240Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8176509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8176669Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8176672Z 
2025-09-07T08:16:09.8176766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8176959Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8177018Z     return mod(*inputs)
2025-09-07T08:16:09.8177287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8177353Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8177681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8177749Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8177963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8178040Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8178297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8178381Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8178638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8178833Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8178841Z 
2025-09-07T08:16:09.8178917Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8178987Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8179061Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8179164Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8179254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8179444Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8179500Z     return mod(*inputs)
2025-09-07T08:16:09.8179771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8179838Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8180100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8180164Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8180376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8180446Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8180738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8180821Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8181078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8181167Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8181513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8181638Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8181643Z 
2025-09-07T08:16:09.8181736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8181920Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8181980Z     return mod(*inputs)
2025-09-07T08:16:09.8182243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8182309Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8182572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8182637Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8182854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8182921Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8183182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8183260Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8183516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8183609Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8183897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8183999Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8184003Z 
2025-09-07T08:16:09.8184090Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8184275Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8184334Z     return mod(*inputs)
2025-09-07T08:16:09.8184635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8184703Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8184960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8185073Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8185289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8185356Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8185617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8185694Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8185959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8186037Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8186040Z 
2025-09-07T08:16:09.8186129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8186316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8186373Z     return mod(*inputs)
2025-09-07T08:16:09.8186672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8186736Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8186994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8187059Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8187305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8187377Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8187635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8187741Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8187748Z 
2025-09-07T08:16:09.8187839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8188025Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8188083Z     return mod(*inputs)
2025-09-07T08:16:09.8188342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8188409Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8188668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8188729Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8188944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8189010Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8189272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8189376Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8189582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8189652Z     return self.act(input)
2025-09-07T08:16:09.8189655Z 
2025-09-07T08:16:09.8189744Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8189934Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8189989Z     return mod(*inputs)
2025-09-07T08:16:09.8190249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8190351Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8190611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8190682Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8190943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8191026Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8191293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8191371Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8191374Z 
2025-09-07T08:16:09.8191479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8191674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8191742Z     return mod(*inputs)
2025-09-07T08:16:09.8192007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8192074Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8192375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8192444Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8192670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8192740Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8193039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8193124Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8193386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8193495Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8193498Z 
2025-09-07T08:16:09.8193592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8193784Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8193841Z     return mod(*inputs)
2025-09-07T08:16:09.8194104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8194174Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8194435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8194506Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8194718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8194792Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8195052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8195132Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8195397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8195554Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8195557Z 
2025-09-07T08:16:09.8195655Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8195842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8195900Z     return mod(*inputs)
2025-09-07T08:16:09.8196161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8196259Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8196526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8196590Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8196845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8196914Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8197170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8197256Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8197516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8197677Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8197682Z 
2025-09-07T08:16:09.8197757Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8197826Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8197896Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8197966Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8198097Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8198285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8198344Z     return mod(*inputs)
2025-09-07T08:16:09.8198608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8198703Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8198974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8199045Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8199261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8199330Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8199589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8199674Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8199933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8200028Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8200324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8200449Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8200454Z 
2025-09-07T08:16:09.8200552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8200736Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8200794Z     return mod(*inputs)
2025-09-07T08:16:09.8201057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8201131Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8201391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8201455Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8201672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8201738Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8202001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8202115Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8202375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8202466Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8202790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8202902Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8202906Z 
2025-09-07T08:16:09.8202998Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8203340Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8203402Z     return mod(*inputs)
2025-09-07T08:16:09.8203667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8203737Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8203999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8204070Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8204366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8204438Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8204701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8204827Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8205093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8205170Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8205173Z 
2025-09-07T08:16:09.8205267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8205454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8205511Z     return mod(*inputs)
2025-09-07T08:16:09.8205790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8205862Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8206137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8206208Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8206433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8206513Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8206775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8206852Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8206855Z 
2025-09-07T08:16:09.8206956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8207154Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8207222Z     return mod(*inputs)
2025-09-07T08:16:09.8207489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8207560Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8207822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8207890Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8208107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8208226Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8208488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8208603Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8208654Z 
2025-09-07T08:16:09.8208758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8208946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8209005Z     return mod(*inputs)
2025-09-07T08:16:09.8209273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8209336Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8209597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8209663Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8209877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8209949Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8210241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8210350Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8210562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8210632Z     return self.act(input)
2025-09-07T08:16:09.8210635Z 
2025-09-07T08:16:09.8210758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8210943Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8211007Z     return mod(*inputs)
2025-09-07T08:16:09.8211270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8211336Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8211592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8211658Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8211879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8211947Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8212212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8212284Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8212287Z 
2025-09-07T08:16:09.8212377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8212562Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8212619Z     return mod(*inputs)
2025-09-07T08:16:09.8212884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8212949Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8213211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8213273Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8213487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8213561Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8213821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8213945Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8214202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8214311Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8214346Z 
2025-09-07T08:16:09.8214437Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8214621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8214683Z     return mod(*inputs)
2025-09-07T08:16:09.8214944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8215013Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8215271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8215333Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8215548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8215615Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8215913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8215993Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8216252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8216412Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8216416Z 
2025-09-07T08:16:09.8216535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8216722Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8216780Z     return mod(*inputs)
2025-09-07T08:16:09.8217048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8217112Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8217374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8217449Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8217692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8217767Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8218025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8218103Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8218370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8218533Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8218536Z 
2025-09-07T08:16:09.8218612Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8218681Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8218757Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8218824Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8218913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8219101Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8219162Z     return mod(*inputs)
2025-09-07T08:16:09.8219429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8219494Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8219796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8219871Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8220087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8220197Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8220455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8220535Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8220796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8220886Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8221184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8221310Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8221313Z 
2025-09-07T08:16:09.8221406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8221592Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8221684Z     return mod(*inputs)
2025-09-07T08:16:09.8221951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8222015Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8222275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8222374Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8222591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8222666Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8222926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8223009Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8223270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8223363Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8223654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8223757Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8223760Z 
2025-09-07T08:16:09.8223859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8224047Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8224107Z     return mod(*inputs)
2025-09-07T08:16:09.8224374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8224439Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8224704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8224774Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8224994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8225060Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8225327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8225407Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8225664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8225773Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8225776Z 
2025-09-07T08:16:09.8225866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8226055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8226162Z     return mod(*inputs)
2025-09-07T08:16:09.8226426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8226491Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8226752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8226818Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8227033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8227102Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8227363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8227472Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8227478Z 
2025-09-07T08:16:09.8227609Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8227796Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8227856Z     return mod(*inputs)
2025-09-07T08:16:09.8228116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8228212Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8228479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8228545Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8228764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8228831Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8229091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8229196Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8229403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8229473Z     return self.act(input)
2025-09-07T08:16:09.8229476Z 
2025-09-07T08:16:09.8229570Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8229758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8229815Z     return mod(*inputs)
2025-09-07T08:16:09.8230077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8230146Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8230404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8230477Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8230689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8230759Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8231018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8231092Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8231095Z 
2025-09-07T08:16:09.8231190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8231416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8231474Z     return mod(*inputs)
2025-09-07T08:16:09.8231735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8231880Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8232145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8232209Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8232428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8232501Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8232760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8232846Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8233104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8233210Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8233214Z 
2025-09-07T08:16:09.8233302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8233521Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8233580Z     return mod(*inputs)
2025-09-07T08:16:09.8233841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8233912Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8234205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8234271Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8234486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8234551Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8234815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8234894Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8235156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8235309Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8235313Z 
2025-09-07T08:16:09.8235408Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8235591Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8235647Z     return mod(*inputs)
2025-09-07T08:16:09.8235913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8235974Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8236235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8236298Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8236510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8236579Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8236842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8236924Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8237182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8237373Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8237380Z 
2025-09-07T08:16:09.8237451Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8237518Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8237623Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8237692Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8237784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8237987Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8238045Z     return mod(*inputs)
2025-09-07T08:16:09.8238314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8238376Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8238639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8238704Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8238919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8238993Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8239287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8239371Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8239630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8239749Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8240050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8240175Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8240178Z 
2025-09-07T08:16:09.8240276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8240461Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8240525Z     return mod(*inputs)
2025-09-07T08:16:09.8240789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8240856Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8241122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8241189Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8241408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8241476Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8241735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8241819Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8242084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8242176Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8242463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8242566Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8242568Z 
2025-09-07T08:16:09.8242661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8242853Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8242955Z     return mod(*inputs)
2025-09-07T08:16:09.8243218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8243286Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8243546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8243645Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8243863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8243929Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8244197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8244274Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8244541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8244615Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8244618Z 
2025-09-07T08:16:09.8244706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8244897Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8244988Z     return mod(*inputs)
2025-09-07T08:16:09.8245262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8245327Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8245584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8245681Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8245895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8245965Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8246223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8246294Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8246299Z 
2025-09-07T08:16:09.8246392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8246575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8246637Z     return mod(*inputs)
2025-09-07T08:16:09.8246897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8246967Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8247317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8247415Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8247676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8253879Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8254186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8254320Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8254325Z 
2025-09-07T08:16:09.8254427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8254644Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8254706Z     return mod(*inputs)
2025-09-07T08:16:09.8255001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8255078Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8255424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8255496Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8255721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8255840Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8256113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8256231Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8256450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8256518Z     return self.act(input)
2025-09-07T08:16:09.8256522Z 
2025-09-07T08:16:09.8256629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8256833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8256892Z     return mod(*inputs)
2025-09-07T08:16:09.8257165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8257234Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8257620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8257688Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8257909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8257988Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8258290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8258371Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8258378Z 
2025-09-07T08:16:09.8258474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8258670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8258730Z     return mod(*inputs)
2025-09-07T08:16:09.8258994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8259064Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8259324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8259398Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8259625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8259702Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8259976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8260065Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8260329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8260440Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8260444Z 
2025-09-07T08:16:09.8260549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8260746Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8260807Z     return mod(*inputs)
2025-09-07T08:16:09.8261081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8261151Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8261419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8261523Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8261743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8261819Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8262118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8262206Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8262475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8262649Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8262655Z 
2025-09-07T08:16:09.8262753Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8262945Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8263010Z     return mod(*inputs)
2025-09-07T08:16:09.8263275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8263345Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8263639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8263708Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8263927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8263997Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8264292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8264372Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8264632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8264792Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8264796Z 
2025-09-07T08:16:09.8264871Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8264944Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8265012Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8265082Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8265174Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8265362Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8265423Z     return mod(*inputs)
2025-09-07T08:16:09.8265698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8265769Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8266042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8266109Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8266337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8266410Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8266676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8266755Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8267014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8267108Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8267442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8267572Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8267576Z 
2025-09-07T08:16:09.8267674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8267926Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8267985Z     return mod(*inputs)
2025-09-07T08:16:09.8268251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8268321Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8268582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8268648Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8268864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8268936Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8269194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8269275Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8269567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8269654Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8269945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8270077Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8270081Z 
2025-09-07T08:16:09.8270173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8270370Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8270429Z     return mod(*inputs)
2025-09-07T08:16:09.8270694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8270758Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8271018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8271085Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8271298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8271374Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8271632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8271720Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8271981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8272053Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8272056Z 
2025-09-07T08:16:09.8272152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8272341Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8272402Z     return mod(*inputs)
2025-09-07T08:16:09.8272664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8272731Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8272992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8273057Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8273313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8273384Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8273641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8273796Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8273799Z 
2025-09-07T08:16:09.8273891Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8274082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8274141Z     return mod(*inputs)
2025-09-07T08:16:09.8274423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8274491Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8274765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8274833Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8275050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8275122Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8275414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8275524Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8275737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8275833Z     return self.act(input)
2025-09-07T08:16:09.8275837Z 
2025-09-07T08:16:09.8275935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8276130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8276192Z     return mod(*inputs)
2025-09-07T08:16:09.8276455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8276519Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8276781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8276847Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8277061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8277127Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8277384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8277460Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8277465Z 
2025-09-07T08:16:09.8277557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8277744Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8277805Z     return mod(*inputs)
2025-09-07T08:16:09.8278065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8278136Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8278397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8278467Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8278682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8278754Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8279015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8279133Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8279398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8279507Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8279545Z 
2025-09-07T08:16:09.8279645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8279832Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8279892Z     return mod(*inputs)
2025-09-07T08:16:09.8280162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8280230Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8280502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8280570Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8280790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8280860Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8281156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8281243Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8281504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8281698Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8281703Z 
2025-09-07T08:16:09.8281796Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8281983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8282050Z     return mod(*inputs)
2025-09-07T08:16:09.8282314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8282379Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8282639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8282704Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8282916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8282982Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8283242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8283318Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8283578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8283731Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8283735Z 
2025-09-07T08:16:09.8283807Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8283881Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8283950Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8284020Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8284108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8284289Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8284349Z     return mod(*inputs)
2025-09-07T08:16:09.8284615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8284719Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8284977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8285046Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8285261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8285364Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8285625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8285704Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8285972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8286063Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8286359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8286490Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8286493Z 
2025-09-07T08:16:09.8286586Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8286807Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8286866Z     return mod(*inputs)
2025-09-07T08:16:09.8287132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8287198Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8287488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8287559Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8287773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8287849Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8288109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8288186Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8288451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8288536Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8288830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8288930Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8288934Z 
2025-09-07T08:16:09.8289029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8289213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8289273Z     return mod(*inputs)
2025-09-07T08:16:09.8289536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8289600Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8289867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8289931Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8290143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8290214Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8290476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8290560Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8290870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8290942Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8290949Z 
2025-09-07T08:16:09.8291038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8291258Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8291324Z     return mod(*inputs)
2025-09-07T08:16:09.8291585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8291654Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8291915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8291981Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8292199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8292267Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8292533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8292607Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8292643Z 
2025-09-07T08:16:09.8292735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8292921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8292980Z     return mod(*inputs)
2025-09-07T08:16:09.8293275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8293342Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8293610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8293679Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8293891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8293964Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8294226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8294337Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8294341Z 
2025-09-07T08:16:09.8294430Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8294612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8294679Z     return mod(*inputs)
2025-09-07T08:16:09.8294943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8295015Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8295273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8295341Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8295557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8295624Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8295885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8295988Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8296203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8296267Z     return self.act(input)
2025-09-07T08:16:09.8296270Z 
2025-09-07T08:16:09.8296396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8296586Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8296643Z     return mod(*inputs)
2025-09-07T08:16:09.8296911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8297010Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8297268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8297330Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8297584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8297655Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8297919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8297997Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8298001Z 
2025-09-07T08:16:09.8298094Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8298279Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8298339Z     return mod(*inputs)
2025-09-07T08:16:09.8298642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8298709Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8298976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8299073Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8299290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8299366Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8299624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8299707Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8299968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8300071Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8300080Z 
2025-09-07T08:16:09.8300173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8300357Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8300420Z     return mod(*inputs)
2025-09-07T08:16:09.8300682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8300756Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8301015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8301081Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8301298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8301368Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8301634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8301712Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8301971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8302132Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8302135Z 
2025-09-07T08:16:09.8302260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8302450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8302507Z     return mod(*inputs)
2025-09-07T08:16:09.8302775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8303254Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8303516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8303583Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8303799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8303872Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8304133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8304214Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8304478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8304635Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8304704Z 
2025-09-07T08:16:09.8304785Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8304853Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8304928Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8305002Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8305096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8305334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8305396Z     return mod(*inputs)
2025-09-07T08:16:09.8305663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8305735Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8305998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8306071Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8306289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8306364Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8306623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8306709Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8306975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8307066Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8307363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8307488Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8307493Z 
2025-09-07T08:16:09.8307596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8307780Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8307840Z     return mod(*inputs)
2025-09-07T08:16:09.8308110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8308177Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8308439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8308556Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8308772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8308851Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8309114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8309239Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8309502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8309585Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8309883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8309986Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8309990Z 
2025-09-07T08:16:09.8310087Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8310271Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8310332Z     return mod(*inputs)
2025-09-07T08:16:09.8310633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8310701Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8310965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8311030Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8311291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8311362Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8311622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8311708Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8311964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8312040Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8312045Z 
2025-09-07T08:16:09.8312137Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8312323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8312379Z     return mod(*inputs)
2025-09-07T08:16:09.8312640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8312706Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8312962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8313034Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8313247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8313315Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8313581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8313692Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8313695Z 
2025-09-07T08:16:09.8313797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8313981Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8314044Z     return mod(*inputs)
2025-09-07T08:16:09.8314307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8314405Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8314671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8314738Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8314957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8315054Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8315315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8315423Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8315633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8315699Z     return self.act(input)
2025-09-07T08:16:09.8315703Z 
2025-09-07T08:16:09.8315793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8315985Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8316043Z     return mod(*inputs)
2025-09-07T08:16:09.8316304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8316409Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8316673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8316742Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8316957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8317055Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8317319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8317397Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8317401Z 
2025-09-07T08:16:09.8317494Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8317679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8317737Z     return mod(*inputs)
2025-09-07T08:16:09.8318007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8318071Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8318332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8318394Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8318614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8318683Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8318946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8319028Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8319287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T08:16:09.8319396Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T08:16:09.8319400Z 
2025-09-07T08:16:09.8319491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8319674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8319735Z     return mod(*inputs)
2025-09-07T08:16:09.8319996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8320065Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8320358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8320427Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8320639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8320745Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8321010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8321090Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8321352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T08:16:09.8321506Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8321511Z 
2025-09-07T08:16:09.8321601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8321797Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8321859Z     return mod(*inputs)
2025-09-07T08:16:09.8322136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8322268Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8322538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8322607Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8322822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8322930Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8323193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8323283Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8323544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T08:16:09.8323705Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T08:16:09.8323710Z 
2025-09-07T08:16:09.8323793Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8323862Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8323937Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8324006Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8324100Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8324290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8324354Z     return mod(*inputs)
2025-09-07T08:16:09.8324620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8324685Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8324946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8325013Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8325229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8325309Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8325568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8325652Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8325909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8325997Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8326334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T08:16:09.8326461Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T08:16:09.8326465Z 
2025-09-07T08:16:09.8326564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8326788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8326854Z     return mod(*inputs)
2025-09-07T08:16:09.8327115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8327180Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8327447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8327515Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8327735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8327804Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8328062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8328182Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8328440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T08:16:09.8328530Z     attn_output, attn_weights = attention_interface(
2025-09-07T08:16:09.8328852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T08:16:09.8328955Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T08:16:09.8328964Z 
2025-09-07T08:16:09.8329059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8329246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8329311Z     return mod(*inputs)
2025-09-07T08:16:09.8329572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8329643Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8329902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8329968Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8330190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8330260Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8330521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T08:16:09.8330604Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T08:16:09.8330863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T08:16:09.8330941Z     attn_output = self.out_proj(attn_output)
2025-09-07T08:16:09.8330946Z 
2025-09-07T08:16:09.8331037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8331226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8331283Z     return mod(*inputs)
2025-09-07T08:16:09.8331549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8331613Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8331872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8331977Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8332191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8332266Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8332527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T08:16:09.8332633Z     hidden_states = residual + hidden_states
2025-09-07T08:16:09.8332636Z 
2025-09-07T08:16:09.8332732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8332915Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8332977Z     return mod(*inputs)
2025-09-07T08:16:09.8333240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8333308Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8333567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8333631Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8333855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8333973Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8334241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8334354Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8334357Z 
2025-09-07T08:16:09.8334447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8334665Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8334723Z     return mod(*inputs)
2025-09-07T08:16:09.8334993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8335059Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8335319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8335392Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8335607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8335678Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8335938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T08:16:09.8336046Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T08:16:09.8336253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T08:16:09.8336318Z     return self.act(input)
2025-09-07T08:16:09.8336321Z 
2025-09-07T08:16:09.8336421Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8336604Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8336667Z     return mod(*inputs)
2025-09-07T08:16:09.8336931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T08:16:09.8336996Z     encoder_outputs = self.encoder(
2025-09-07T08:16:09.8337263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T08:16:09.8337326Z     layer_outputs = encoder_layer(
2025-09-07T08:16:09.8337613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T08:16:09.8337683Z     return super().__call__(*args, **kwargs)
2025-09-07T08:16:09.8337983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T08:16:09.8338059Z     hidden_states = self.fc2(hidden_states)
2025-09-07T08:16:09.8338063Z 
2025-09-07T08:16:09.8338154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8338348Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8338439Z     return mod(*inputs)
2025-09-07T08:16:09.8338705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1612, in forward
2025-09-07T08:16:09.8338785Z     hidden_states = self.projector(hidden_states)
2025-09-07T08:16:09.8338788Z 
2025-09-07T08:16:09.8338863Z cudagraph partition due to non gpu ops
2025-09-07T08:16:09.8338955Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8339138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8339201Z     return mod(*inputs)
2025-09-07T08:16:09.8339459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1613, in forward
2025-09-07T08:16:09.8339539Z     pooled_output = hidden_states.mean(dim=1)
2025-09-07T08:16:09.8339549Z 
2025-09-07T08:16:09.8339675Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:09.8339860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:09.8339922Z     return mod(*inputs)
2025-09-07T08:16:09.8340181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1615, in forward
2025-09-07T08:16:09.8340263Z     logits = self.classifier(pooled_output)
2025-09-07T08:16:09.8340297Z 
2025-09-07T08:16:20.4040470Z pass
2025-09-07T08:16:20.4040879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:23.2809711Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:16:23.2810613Z   import pynvml  # type: ignore[import]
2025-09-07T08:16:25.2933515Z 
2025-09-07T08:16:25.4455816Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:16:25.4456083Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:16:25.4458158Z cpu  eval  lennard_jones                      
2025-09-07T08:16:25.4471091Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:25.4494203Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:25.4509529Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:27.3264490Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3264939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3265323Z     return mod(*inputs)
2025-09-07T08:16:27.3265435Z 
2025-09-07T08:16:27.3265558Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3265962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3266292Z     return mod(*inputs)
2025-09-07T08:16:27.3266403Z 
2025-09-07T08:16:27.3266505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3266885Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3267231Z     return mod(*inputs)
2025-09-07T08:16:27.3267361Z 
2025-09-07T08:16:27.3267466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3267830Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3268501Z     return mod(*inputs)
2025-09-07T08:16:27.3268598Z 
2025-09-07T08:16:27.3268704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3269049Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3269471Z     return mod(*inputs)
2025-09-07T08:16:27.3269571Z 
2025-09-07T08:16:27.3269668Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3270013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3270328Z     return mod(*inputs)
2025-09-07T08:16:27.3270431Z 
2025-09-07T08:16:27.3270531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3270891Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3271217Z     return mod(*inputs)
2025-09-07T08:16:27.3271311Z 
2025-09-07T08:16:27.3271414Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3271750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3272071Z     return mod(*inputs)
2025-09-07T08:16:27.3272165Z 
2025-09-07T08:16:27.3272261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T08:16:27.3272709Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:16:27.3273040Z     return mod(*inputs)
2025-09-07T08:16:27.3273139Z 
2025-09-07T08:16:31.5218480Z pass
2025-09-07T08:16:31.5220260Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:33.0244492Z accuracy            pass_rate=91.30%
2025-09-07T08:16:33.0248602Z calls_captured      gmean=0.00x mean=416.435x
2025-09-07T08:16:33.0251255Z unique_graphs       gmean=0.00x mean=3.000x
2025-09-07T08:16:33.0253533Z graph_breaks        gmean=0.00x mean=2.174x
2025-09-07T08:16:33.0255772Z unique_graph_breaks gmean=0.00x mean=0.565x
2025-09-07T08:16:33.0257977Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T08:16:33.0260101Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T08:16:33.0262173Z cudagraph_skips     gmean=0.00x mean=2.435x
2025-09-07T08:16:33.0263033Z compilation_latency mean=23.057 seconds
2025-09-07T08:16:33.5462490Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *cppwrapper-true* ]]
2025-09-07T08:16:33.5462991Z + TORCHINDUCTOR_CPP_WRAPPER=1
2025-09-07T08:16:33.5463953Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv
2025-09-07T08:16:33.9420744Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:16:33.9421596Z   import pynvml  # type: ignore[import]
2025-09-07T08:16:36.3836622Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:16:36.3837521Z   import pynvml  # type: ignore[import]
2025-09-07T08:16:38.4053482Z 
2025-09-07T08:16:43.6439312Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:16:43.6439649Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:16:43.6720873Z cpu  eval  dlrm                               
2025-09-07T08:16:43.7998485Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:43.8246854Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:43.8503872Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:55.0486633Z pass
2025-09-07T08:16:55.0488622Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:16:57.2445476Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:16:57.2446604Z   import pynvml  # type: ignore[import]
2025-09-07T08:16:59.2551329Z 
2025-09-07T08:17:01.9431538Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:17:02.5709082Z 
2025-09-07T08:17:02.5709607Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:17:02.5710011Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:17:02.5710414Z cpu  eval  doctr_det_predictor                
2025-09-07T08:17:17.1846392Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:17:17.1847326Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:17:18.0914109Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:17:18.9938880Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:17:50.1677520Z pass
2025-09-07T08:17:50.1682250Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:17:53.4743687Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:17:53.4744701Z   import pynvml  # type: ignore[import]
2025-09-07T08:17:55.4822340Z 
2025-09-07T08:17:58.1519783Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:17:58.1956444Z 
2025-09-07T08:17:58.1956785Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:17:58.1957216Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:17:58.1958086Z cpu  eval  doctr_reco_predictor               
2025-09-07T08:17:58.6806269Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:17:58.6806841Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:17:58.7417769Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:17:58.7898380Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:11.7831695Z pass
2025-09-07T08:18:11.7836176Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:14.0377732Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:18:14.0378618Z   import pynvml  # type: ignore[import]
2025-09-07T08:18:16.0371641Z 
2025-09-07T08:18:16.0898904Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T08:18:16.0899973Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T08:18:16.0900582Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T08:18:16.0901175Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T08:18:16.9767255Z 
2025-09-07T08:18:16.9767691Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:18:16.9782752Z cpu  eval  drq                                
2025-09-07T08:18:16.9958442Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:17.0061529Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:17.0135689Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:26.0446735Z pass
2025-09-07T08:18:26.0447131Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:28.0956929Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:18:28.0957841Z   import pynvml  # type: ignore[import]
2025-09-07T08:18:30.1174330Z 
2025-09-07T08:18:31.7243720Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:18:31.7244039Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:18:31.7427205Z cpu  eval  fastNLP_Bert                       
2025-09-07T08:18:32.7436864Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:33.1387317Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:33.5262712Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:33.6140962Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:18:33.6141679Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:18:33.6142281Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T08:18:33.6142764Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:18:33.6143351Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T08:18:33.6143833Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:18:33.6144295Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T08:18:33.6144970Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:18:33.6145624Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(*inputs)
2025-09-07T08:18:33.6146273Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T08:18:33.6146946Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     sequence_output = self.bert(words)
2025-09-07T08:18:33.6147657Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T08:18:33.6148702Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.model(words)
2025-09-07T08:18:33.6149408Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T08:18:33.6150770Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T08:18:33.6151367Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:18:33.6151750Z W0907 08:18:33.613451 68348 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:18:53.5345870Z pass
2025-09-07T08:18:53.5349208Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:56.0726570Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:18:56.0727697Z   import pynvml  # type: ignore[import]
2025-09-07T08:18:58.0833490Z 
2025-09-07T08:18:58.5146635Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:18:58.5146935Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:18:58.5180141Z cpu  eval  functorch_dp_cifar10               
2025-09-07T08:18:58.5854916Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:58.6124178Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:18:58.6361458Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:11.7422220Z pass
2025-09-07T08:19:11.7424584Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:13.9312655Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:19:13.9313827Z   import pynvml  # type: ignore[import]
2025-09-07T08:19:15.9435138Z 
2025-09-07T08:19:16.1359876Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:19:16.1360169Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:19:16.1360438Z cpu  eval  functorch_maml_omniglot            
2025-09-07T08:19:16.1464147Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:16.1541375Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:16.1588731Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:24.6099789Z pass
2025-09-07T08:19:24.6102075Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:26.5856713Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:19:26.5857631Z   import pynvml  # type: ignore[import]
2025-09-07T08:19:28.6000579Z 
2025-09-07T08:19:30.2791902Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:19:30.2792377Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:19:30.2842126Z cpu  eval  hf_Albert                          
2025-09-07T08:19:31.2003558Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:31.4338149Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:31.7299460Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:46.7909985Z pass
2025-09-07T08:19:46.7910679Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:49.3119793Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:19:49.3120653Z   import pynvml  # type: ignore[import]
2025-09-07T08:19:51.3269898Z 
2025-09-07T08:19:55.4931451Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:19:55.4932707Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:19:55.5159829Z cpu  eval  hf_Bart                            
2025-09-07T08:19:56.9244223Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:57.2758791Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:19:57.6244860Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:19.5499114Z pass
2025-09-07T08:20:19.5499514Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:22.4469439Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:20:22.4470326Z   import pynvml  # type: ignore[import]
2025-09-07T08:20:24.4715413Z 
2025-09-07T08:20:27.5908612Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:20:27.5908928Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:20:27.6145578Z cpu  eval  hf_Bert                            
2025-09-07T08:20:28.7199638Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:29.0044966Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:29.2825978Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:46.3649530Z pass
2025-09-07T08:20:46.3649936Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:49.1225631Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:20:49.1226499Z   import pynvml  # type: ignore[import]
2025-09-07T08:20:51.1318573Z 
2025-09-07T08:20:56.4991772Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:20:56.4992094Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:20:56.5398449Z cpu  eval  hf_Bert_large                      
2025-09-07T08:20:58.8700592Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:20:59.5192138Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:21:00.1590353Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:21:28.6826724Z pass
2025-09-07T08:21:28.6827123Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:21:31.7273732Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:21:31.7275073Z   import pynvml  # type: ignore[import]
2025-09-07T08:21:33.7340128Z 
2025-09-07T08:21:35.1280850Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:21:35.1282185Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:21:35.1283004Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:21:35.1283659Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:21:35.1285309Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:21:35.1286687Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:21:35.1287512Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:21:35.1288155Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:21:36.9583114Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:21:36.9583922Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:21:38.8402477Z 
2025-09-07T08:21:38.8403809Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:21:38.8632969Z cpu  eval  hf_BigBird                         
2025-09-07T08:21:41.3858395Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:21:44.4333237Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:21:47.4215069Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:18.3015505Z W0907 08:22:18.300029 76432 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] torch._dynamo hit config.recompile_limit (8)
2025-09-07T08:22:18.3016461Z W0907 08:22:18.300029 76432 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    function: 'forward' (/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py:417)
2025-09-07T08:22:18.3018379Z W0907 08:22:18.300029 76432 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    last reason: 10/7: self.seed == 7                                           # context_layer, attention_probs = self.bigbird_block_sparse_attention(  # transformers/models/big_bird/modeling_big_bird.py:455 in forward (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
2025-09-07T08:22:18.3020489Z W0907 08:22:18.300029 76432 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-09-07T08:22:18.3021282Z W0907 08:22:18.300029 76432 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2025-09-07T08:22:23.7943515Z pass
2025-09-07T08:22:23.7943902Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:26.9062239Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:22:26.9063148Z   import pynvml  # type: ignore[import]
2025-09-07T08:22:28.9265988Z 
2025-09-07T08:22:31.5187188Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:22:31.5187522Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:22:31.5318948Z cpu  eval  hf_DistilBert                      
2025-09-07T08:22:32.1196178Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:32.2821009Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:32.4345785Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:44.8162669Z pass
2025-09-07T08:22:44.8163072Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:47.3707842Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:22:47.3708746Z   import pynvml  # type: ignore[import]
2025-09-07T08:22:49.3874184Z 
2025-09-07T08:22:54.0575934Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:22:54.0576334Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:22:54.0752338Z cpu  eval  hf_GPT2                            
2025-09-07T08:22:54.6321179Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:54.7957659Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:22:54.9599357Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:23:12.1194164Z pass
2025-09-07T08:23:12.1194588Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:23:14.8965698Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:23:14.8966726Z   import pynvml  # type: ignore[import]
2025-09-07T08:23:16.9138560Z 
2025-09-07T08:23:31.1541606Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:23:31.1541942Z loading model: 0it [00:14, ?it/s]
2025-09-07T08:23:31.2149438Z cpu  eval  hf_GPT2_large                      
2025-09-07T08:23:31.6070923Z pass_due_to_skip
2025-09-07T08:23:31.6075261Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:23:34.0089216Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:23:34.0090412Z   import pynvml  # type: ignore[import]
2025-09-07T08:23:36.0233091Z 
2025-09-07T08:23:40.0757813Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:23:40.0758107Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:23:40.1051988Z cpu  eval  hf_Longformer                      
2025-09-07T08:23:41.8437702Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:23:42.6335270Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:23:43.3739727Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:23:43.5444359Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:23:43.5445021Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:23:43.5445548Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T08:23:43.5446054Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:23:43.5447018Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T08:23:43.5447513Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:23:43.5447964Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T08:23:43.5448728Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:23:43.5449401Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(**inputs)
2025-09-07T08:23:43.5450174Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T08:23:43.5450938Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.longformer(
2025-09-07T08:23:43.5451726Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T08:23:43.5452484Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     encoder_outputs = self.encoder(
2025-09-07T08:23:43.5453223Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1244, in forward
2025-09-07T08:23:43.5454018Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2025-09-07T08:23:43.5454512Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:23:43.5454881Z W0907 08:23:43.543764 81358 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:25:23.7218915Z pass
2025-09-07T08:25:23.7219309Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:25:27.7222502Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:25:27.7223729Z   import pynvml  # type: ignore[import]
2025-09-07T08:25:29.7373845Z 
2025-09-07T08:25:31.1147093Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:25:31.1149437Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:25:31.8694205Z 
2025-09-07T08:25:31.8694624Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:25:31.8744925Z cpu  eval  hf_Reformer                        
2025-09-07T08:25:32.4035723Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:25:33.8553986Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:25:35.3094055Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:25:42.5542031Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:25:42.5543276Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:25:42.5543813Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] or:
2025-09-07T08:25:42.5544430Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:25:42.5545006Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] to include these operations in the captured graph.
2025-09-07T08:25:42.5545490Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:25:42.5545955Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break: from user code at:
2025-09-07T08:25:42.5546795Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2132, in torch_dynamo_resume_in_forward_at_2104
2025-09-07T08:25:42.5547669Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     embedding_output = self.embeddings(
2025-09-07T08:25:42.5548416Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 369, in forward
2025-09-07T08:25:42.5549250Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     position_embeddings = self.position_embeddings(position_ids)
2025-09-07T08:25:42.5550075Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T08:25:42.5550847Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     max_position_id = position_ids.max().item()
2025-09-07T08:25:42.5551315Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:25:42.5551677Z W0907 08:25:42.553452 82806 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:26:00.6097144Z pass
2025-09-07T08:26:00.6097556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:03.2110060Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:26:03.2113594Z   import pynvml  # type: ignore[import]
2025-09-07T08:26:05.2210022Z 
2025-09-07T08:26:10.9653827Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:26:10.9655356Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:26:11.0047545Z cpu  eval  hf_Roberta_base                    
2025-09-07T08:26:12.9226074Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:13.4540958Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:13.9850728Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:34.1059223Z pass
2025-09-07T08:26:34.1059641Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:37.7535363Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:26:37.7536643Z   import pynvml  # type: ignore[import]
2025-09-07T08:26:39.7797018Z 
2025-09-07T08:26:42.6228388Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:26:42.6228673Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:26:42.6438561Z cpu  eval  hf_T5                              
2025-09-07T08:26:43.6408137Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:44.0964723Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:26:44.5426152Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:27:10.0498905Z pass
2025-09-07T08:27:10.0499288Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:27:12.9431824Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:27:12.9432690Z   import pynvml  # type: ignore[import]
2025-09-07T08:27:14.9568074Z 
2025-09-07T08:27:24.5063370Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:27:24.5063653Z loading model: 0it [00:09, ?it/s]
2025-09-07T08:27:24.5469624Z cpu  eval  hf_T5_base                         
2025-09-07T08:27:39.0921829Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:27:45.7528039Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:27:52.2914291Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:28:55.2393670Z pass
2025-09-07T08:28:55.2394085Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:29:01.9380918Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:29:01.9382055Z   import pynvml  # type: ignore[import]
2025-09-07T08:29:03.9647178Z 
2025-09-07T08:29:16.4534692Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:29:16.4535435Z loading model: 0it [00:12, ?it/s]
2025-09-07T08:29:16.5360029Z cpu  eval  hf_T5_large                        
2025-09-07T08:29:16.9380754Z pass_due_to_skip
2025-09-07T08:29:16.9383985Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:29:19.2481891Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:29:19.2483073Z   import pynvml  # type: ignore[import]
2025-09-07T08:29:21.2658390Z 
2025-09-07T08:29:27.4875408Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:29:27.4875700Z loading model: 0it [00:06, ?it/s]
2025-09-07T08:29:27.5248830Z cpu  eval  hf_distil_whisper                  
2025-09-07T08:29:34.8381690Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:29:36.9567906Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:29:39.0777093Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:15.0977429Z pass
2025-09-07T08:30:15.0977888Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:18.3866224Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:30:18.3867132Z   import pynvml  # type: ignore[import]
2025-09-07T08:30:20.3856909Z 
2025-09-07T08:30:20.5341989Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:30:20.5342275Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:30:20.5346795Z cpu  eval  lennard_jones                      
2025-09-07T08:30:20.5355920Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:20.5381722Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:20.5396247Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:27.6310332Z pass
2025-09-07T08:30:27.6312414Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:29.1302107Z accuracy            pass_rate=91.30%
2025-09-07T08:30:29.1307305Z calls_captured      gmean=0.00x mean=411.261x
2025-09-07T08:30:29.1310316Z unique_graphs       gmean=0.00x mean=2.957x
2025-09-07T08:30:29.1312615Z graph_breaks        gmean=0.00x mean=2.174x
2025-09-07T08:30:29.1314701Z unique_graph_breaks gmean=0.00x mean=0.565x
2025-09-07T08:30:29.1316800Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T08:30:29.1318825Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T08:30:29.1320879Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T08:30:29.1321691Z compilation_latency mean=22.077 seconds
2025-09-07T08:30:29.6622584Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *freezing_cudagraphs-true* ]]
2025-09-07T08:30:29.6623406Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *freeze_autotune_cudagraphs-true* ]]
2025-09-07T08:30:29.6624171Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *aotinductor-true* ]]
2025-09-07T08:30:29.6624629Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2025-09-07T08:30:29.6624837Z + [[ accuracy == \a\c\c\u\r\a\c\y ]]
2025-09-07T08:30:29.6625745Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --export --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_export_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv
2025-09-07T08:30:30.0572363Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:30:30.0573570Z   import pynvml  # type: ignore[import]
2025-09-07T08:30:32.4980883Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:30:32.4981767Z   import pynvml  # type: ignore[import]
2025-09-07T08:30:34.5124074Z 
2025-09-07T08:30:39.8422003Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:30:39.8422391Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:30:39.8701492Z cpu  eval  dlrm                               
2025-09-07T08:30:39.9976309Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:40.0225500Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:40.0483724Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:45.5746905Z pass
2025-09-07T08:30:45.5748884Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:30:46.9542872Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:30:46.9544272Z   import pynvml  # type: ignore[import]
2025-09-07T08:30:48.9749226Z 
2025-09-07T08:30:51.6794760Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:30:52.3054419Z 
2025-09-07T08:30:52.3059413Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:30:52.3059838Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:30:52.3060183Z cpu  eval  doctr_det_predictor                
2025-09-07T08:31:06.8494342Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:06.8494874Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:31:07.7525158Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:08.6492414Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:09.7771272Z ERROR:common:
2025-09-07T08:31:09.7771515Z Traceback (most recent call last):
2025-09-07T08:31:09.7771934Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in check_accuracy
2025-09-07T08:31:09.7772300Z     optimized_model_iter_fn = optimize_ctx(
2025-09-07T08:31:09.7772619Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1523, in export
2025-09-07T08:31:09.7772941Z     ep = torch.export.export(
2025-09-07T08:31:09.7773289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 311, in export
2025-09-07T08:31:09.7773627Z     raise e
2025-09-07T08:31:09.7773903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 277, in export
2025-09-07T08:31:09.7774239Z     return _export(
2025-09-07T08:31:09.7774528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:31:09.7774843Z     raise e
2025-09-07T08:31:09.7775112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:31:09.7775424Z     ep = fn(*args, **kwargs)
2025-09-07T08:31:09.7776160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:31:09.7776519Z     return fn(*args, **kwargs)
2025-09-07T08:31:09.7776833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2255, in _export
2025-09-07T08:31:09.7777173Z     ep = _export_for_training(
2025-09-07T08:31:09.7777666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:31:09.7777984Z     raise e
2025-09-07T08:31:09.7778256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:31:09.7778569Z     ep = fn(*args, **kwargs)
2025-09-07T08:31:09.7778899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:31:09.7779248Z     return fn(*args, **kwargs)
2025-09-07T08:31:09.7779591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
2025-09-07T08:31:09.7779956Z     export_artifact = export_func(
2025-09-07T08:31:09.7780287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1415, in _strict_export
2025-09-07T08:31:09.7780643Z     gm_torch_level = _export_to_torch_ir(
2025-09-07T08:31:09.7781092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 812, in _export_to_torch_ir
2025-09-07T08:31:09.7781469Z     gm_torch_level, _ = torch._dynamo.export(
2025-09-07T08:31:09.7781836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 2002, in inner
2025-09-07T08:31:09.7782181Z     result_traced = opt_f(*args, **kwargs)
2025-09-07T08:31:09.7782590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 414, in __call__
2025-09-07T08:31:09.7782943Z     return super().__call__(*args, **kwargs)
2025-09-07T08:31:09.7783327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
2025-09-07T08:31:09.7783699Z     return self._call_impl(*args, **kwargs)
2025-09-07T08:31:09.7784046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:09.7784398Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:09.7784756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 841, in compile_wrapper
2025-09-07T08:31:09.7785169Z     raise e.with_traceback(None) from e.__cause__  # User compiler error
2025-09-07T08:31:09.7785498Z torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped
2025-09-07T08:31:09.7786177Z   Explanation: Dynamo does not know how to trace the builtin `<unknown module>.morphologyEx.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).
2025-09-07T08:31:09.7786987Z   Hint: If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.
2025-09-07T08:31:09.7787902Z   Hint: If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.
2025-09-07T08:31:09.7788551Z 
2025-09-07T08:31:09.7788762Z   Developer debug context: module: <unknown module>, qualname: morphologyEx, skip reason: <missing reason>
2025-09-07T08:31:09.7789034Z 
2025-09-07T08:31:09.7789317Z  For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0007.html
2025-09-07T08:31:09.7789635Z 
2025-09-07T08:31:09.7789701Z from user code:
2025-09-07T08:31:09.7790119Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 211, in forward
2025-09-07T08:31:09.7790710Z     for preds in self.postprocessor(prob_map.detach().cpu().permute((0, 2, 3, 1)).numpy())
2025-09-07T08:31:09.7791166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/core.py", line 90, in __call__
2025-09-07T08:31:09.7791507Z     bin_map = [
2025-09-07T08:31:09.7792563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/core.py", line 91, in <listcomp>
2025-09-07T08:31:09.7792899Z     [
2025-09-07T08:31:09.7793222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/core.py", line 92, in <listcomp>
2025-09-07T08:31:09.7793655Z     cv2.morphologyEx(bmap[..., idx], cv2.MORPH_OPEN, self._opening_kernel)
2025-09-07T08:31:09.7793851Z 
2025-09-07T08:31:09.7794212Z Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
2025-09-07T08:31:09.7794614Z 
2025-09-07T08:31:09.7794760Z TorchDynamo optimized model failed to run because of following error
2025-09-07T08:31:10.0514921Z fail_to_run
2025-09-07T08:31:10.0515277Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:11.9805696Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:31:11.9806589Z   import pynvml  # type: ignore[import]
2025-09-07T08:31:14.0026114Z 
2025-09-07T08:31:16.6193757Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:31:16.6624035Z 
2025-09-07T08:31:16.6624310Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:31:16.6624710Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:31:16.6625111Z cpu  eval  doctr_reco_predictor               
2025-09-07T08:31:17.1446417Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:17.1447265Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:31:17.2054399Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:17.2529831Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:17.8048293Z ERROR:common:
2025-09-07T08:31:17.8048528Z Traceback (most recent call last):
2025-09-07T08:31:17.8049010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/variables/tensor.py", line 1435, in evaluate_expr
2025-09-07T08:31:17.8049428Z     return guard_scalar(self.sym_num)
2025-09-07T08:31:17.8049839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 1535, in guard_scalar
2025-09-07T08:31:17.8050255Z     return guard_int(a)
2025-09-07T08:31:17.8050627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 1732, in guard_int
2025-09-07T08:31:17.8051085Z     return a.node.guard_int("", 0)  # NB: uses Python backtrace
2025-09-07T08:31:17.8051546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/sym_node.py", line 518, in guard_int
2025-09-07T08:31:17.8051907Z     r = self.evaluate()
2025-09-07T08:31:17.8052249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/sym_node.py", line 512, in evaluate
2025-09-07T08:31:17.8052706Z     return self.shape_env.evaluate_sym_node(self, size_oblivious)
2025-09-07T08:31:17.8053178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 7239, in evaluate_sym_node
2025-09-07T08:31:17.8053928Z     return self.evaluate_expr(
2025-09-07T08:31:17.8054310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 7339, in evaluate_expr
2025-09-07T08:31:17.8054713Z     return self._inner_evaluate_expr(
2025-09-07T08:31:17.8055083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/recording.py", line 272, in wrapper
2025-09-07T08:31:17.8055546Z     return retlog(fn(*args, **kwargs))
2025-09-07T08:31:17.8055951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 7362, in _inner_evaluate_expr
2025-09-07T08:31:17.8056362Z     return self._evaluate_expr(
2025-09-07T08:31:17.8056751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 7580, in _evaluate_expr
2025-09-07T08:31:17.8057158Z     raise self._make_data_dependent_error(
2025-09-07T08:31:17.8057779Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: Could not extract specialized integer from data-dependent expression u0 (unhinted: u0).  (Size-like symbols: none)
2025-09-07T08:31:17.8058260Z 
2025-09-07T08:31:17.8058649Z Caused by: decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab)  # doctr/models/recognition/crnn/pytorch.py:79 in <listcomp> (_dynamo/variables/tensor.py:1435 in evaluate_expr)
2025-09-07T08:31:17.8059258Z For more information, run with TORCH_LOGS="dynamic"
2025-09-07T08:31:17.8059607Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0"
2025-09-07T08:31:17.8060015Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:31:17.8060611Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing
2025-09-07T08:31:17.8060955Z 
2025-09-07T08:31:17.8061038Z User Stack (most recent call last):
2025-09-07T08:31:17.8061240Z   (snipped, see stack below for prefix)
2025-09-07T08:31:17.8061641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward
2025-09-07T08:31:17.8062048Z     out["preds"] = self.postprocessor(logits)
2025-09-07T08:31:17.8062439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__
2025-09-07T08:31:17.8062905Z     return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab))
2025-09-07T08:31:17.8063391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path
2025-09-07T08:31:17.8063773Z     words = [
2025-09-07T08:31:17.8064113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in <listcomp>
2025-09-07T08:31:17.8064562Z     decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab)
2025-09-07T08:31:17.8064755Z 
2025-09-07T08:31:17.8064873Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:31:17.8065053Z 
2025-09-07T08:31:17.8065184Z During handling of the above exception, another exception occurred:
2025-09-07T08:31:17.8065378Z 
2025-09-07T08:31:17.8065792Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not extract specialized integer from data-dependent expression u0 (unhinted: u0).  (Size-like symbols: none)
2025-09-07T08:31:17.8066264Z 
2025-09-07T08:31:17.8066641Z Caused by: decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab)  # doctr/models/recognition/crnn/pytorch.py:79 in <listcomp> (_dynamo/variables/tensor.py:1435 in evaluate_expr)
2025-09-07T08:31:17.8067163Z For more information, run with TORCH_LOGS="dynamic"
2025-09-07T08:31:17.8067503Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0"
2025-09-07T08:31:17.8067900Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:31:17.8068468Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing
2025-09-07T08:31:17.8068812Z 
2025-09-07T08:31:17.8068890Z User Stack (most recent call last):
2025-09-07T08:31:17.8069097Z   (snipped, see stack below for prefix)
2025-09-07T08:31:17.8069495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward
2025-09-07T08:31:17.8069935Z     out["preds"] = self.postprocessor(logits)
2025-09-07T08:31:17.8070319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__
2025-09-07T08:31:17.8070788Z     return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab))
2025-09-07T08:31:17.8071278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path
2025-09-07T08:31:17.8071659Z     words = [
2025-09-07T08:31:17.8072003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in <listcomp>
2025-09-07T08:31:17.8072442Z     decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab)
2025-09-07T08:31:17.8072641Z 
2025-09-07T08:31:17.8072756Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:31:17.8073248Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example
2025-09-07T08:31:17.8073585Z 
2025-09-07T08:31:17.8073731Z The above exception was the direct cause of the following exception:
2025-09-07T08:31:17.8073920Z 
2025-09-07T08:31:17.8074003Z Traceback (most recent call last):
2025-09-07T08:31:17.8074352Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in check_accuracy
2025-09-07T08:31:17.8074693Z     optimized_model_iter_fn = optimize_ctx(
2025-09-07T08:31:17.8075004Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1523, in export
2025-09-07T08:31:17.8075313Z     ep = torch.export.export(
2025-09-07T08:31:17.8075636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 311, in export
2025-09-07T08:31:17.8075951Z     raise e
2025-09-07T08:31:17.8076224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 277, in export
2025-09-07T08:31:17.8076543Z     return _export(
2025-09-07T08:31:17.8076828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:31:17.8077132Z     raise e
2025-09-07T08:31:17.8077397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:31:17.8077721Z     ep = fn(*args, **kwargs)
2025-09-07T08:31:17.8078062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:31:17.8078409Z     return fn(*args, **kwargs)
2025-09-07T08:31:17.8078727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2255, in _export
2025-09-07T08:31:17.8079048Z     ep = _export_for_training(
2025-09-07T08:31:17.8079348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:31:17.8079656Z     raise e
2025-09-07T08:31:17.8079918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:31:17.8080232Z     ep = fn(*args, **kwargs)
2025-09-07T08:31:17.8080558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:31:17.8080902Z     return fn(*args, **kwargs)
2025-09-07T08:31:17.8081235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
2025-09-07T08:31:17.8081601Z     export_artifact = export_func(
2025-09-07T08:31:17.8081986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1415, in _strict_export
2025-09-07T08:31:17.8082344Z     gm_torch_level = _export_to_torch_ir(
2025-09-07T08:31:17.8082705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 812, in _export_to_torch_ir
2025-09-07T08:31:17.8083110Z     gm_torch_level, _ = torch._dynamo.export(
2025-09-07T08:31:17.8083461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 2002, in inner
2025-09-07T08:31:17.8083800Z     result_traced = opt_f(*args, **kwargs)
2025-09-07T08:31:17.8084145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 414, in __call__
2025-09-07T08:31:17.8084492Z     return super().__call__(*args, **kwargs)
2025-09-07T08:31:17.8084874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
2025-09-07T08:31:17.8085249Z     return self._call_impl(*args, **kwargs)
2025-09-07T08:31:17.8085599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:17.8085952Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:17.8086305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 841, in compile_wrapper
2025-09-07T08:31:17.8086755Z     raise e.with_traceback(None) from e.__cause__  # User compiler error
2025-09-07T08:31:17.8087125Z torch._dynamo.exc.Unsupported: Unexpected failure during itertools.groupby() iteration
2025-09-07T08:31:17.8087495Z   Explanation: Unexpected failure in invoking function during groupby
2025-09-07T08:31:17.8088073Z   Hint: It may be possible to write Dynamo tracing rules for this code. Please report an issue to PyTorch if you encounter this graph break often and it is causing performance issues.
2025-09-07T08:31:17.8088482Z 
2025-09-07T08:31:17.8088723Z   Developer debug context: call_function ItertoolsVariable(<class 'itertools.groupby'>) [ListVariable(length=32)] {}
2025-09-07T08:31:17.8089030Z 
2025-09-07T08:31:17.8089293Z  For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0115.html
2025-09-07T08:31:17.8089619Z 
2025-09-07T08:31:17.8089680Z from user code:
2025-09-07T08:31:17.8090039Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 224, in forward
2025-09-07T08:31:17.8090441Z     out["preds"] = self.postprocessor(logits)
2025-09-07T08:31:17.8090826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 99, in __call__
2025-09-07T08:31:17.8091298Z     return self.ctc_best_path(logits=logits, vocab=self.vocab, blank=len(self.vocab))
2025-09-07T08:31:17.8091775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 78, in ctc_best_path
2025-09-07T08:31:17.8092158Z     words = [
2025-09-07T08:31:17.8092497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 79, in <listcomp>
2025-09-07T08:31:17.8092945Z     decode_sequence([k for k, _ in groupby(seq.tolist()) if k != blank], vocab)
2025-09-07T08:31:17.8093143Z 
2025-09-07T08:31:17.8093492Z Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
2025-09-07T08:31:17.8093898Z 
2025-09-07T08:31:17.8094034Z TorchDynamo optimized model failed to run because of following error
2025-09-07T08:31:17.8704525Z fail_to_run
2025-09-07T08:31:17.8705301Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:19.1807157Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:31:19.1808335Z   import pynvml  # type: ignore[import]
2025-09-07T08:31:21.1906229Z 
2025-09-07T08:31:21.2427920Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T08:31:21.2428948Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T08:31:21.2429553Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T08:31:21.2430159Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T08:31:22.1373929Z 
2025-09-07T08:31:22.1374341Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:22.1389027Z cpu  eval  drq                                
2025-09-07T08:31:22.1565645Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:22.1674720Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:22.1752793Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:27.0083198Z pass
2025-09-07T08:31:27.0083817Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:28.2223612Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:31:28.2224506Z   import pynvml  # type: ignore[import]
2025-09-07T08:31:30.2383836Z 
2025-09-07T08:31:31.7919237Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:31.7919575Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:31:31.8097714Z cpu  eval  fastNLP_Bert                       
2025-09-07T08:31:32.8166356Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:33.2079154Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:33.5911233Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:33.9041835Z ERROR:common:
2025-09-07T08:31:33.9042063Z Traceback (most recent call last):
2025-09-07T08:31:33.9042431Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in check_accuracy
2025-09-07T08:31:33.9042803Z     optimized_model_iter_fn = optimize_ctx(
2025-09-07T08:31:33.9043144Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1523, in export
2025-09-07T08:31:33.9043465Z     ep = torch.export.export(
2025-09-07T08:31:33.9043827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 311, in export
2025-09-07T08:31:33.9044177Z     raise e
2025-09-07T08:31:33.9044470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 277, in export
2025-09-07T08:31:33.9044784Z     return _export(
2025-09-07T08:31:33.9045093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:31:33.9045412Z     raise e
2025-09-07T08:31:33.9045683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:31:33.9046023Z     ep = fn(*args, **kwargs)
2025-09-07T08:31:33.9046376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:31:33.9046740Z     return fn(*args, **kwargs)
2025-09-07T08:31:33.9047045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2255, in _export
2025-09-07T08:31:33.9047667Z     ep = _export_for_training(
2025-09-07T08:31:33.9047966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:31:33.9048276Z     raise e
2025-09-07T08:31:33.9048545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:31:33.9048989Z     ep = fn(*args, **kwargs)
2025-09-07T08:31:33.9049310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:31:33.9049678Z     return fn(*args, **kwargs)
2025-09-07T08:31:33.9050037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
2025-09-07T08:31:33.9050408Z     export_artifact = export_func(
2025-09-07T08:31:33.9050745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1415, in _strict_export
2025-09-07T08:31:33.9051099Z     gm_torch_level = _export_to_torch_ir(
2025-09-07T08:31:33.9051455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 812, in _export_to_torch_ir
2025-09-07T08:31:33.9051838Z     gm_torch_level, _ = torch._dynamo.export(
2025-09-07T08:31:33.9052188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 2002, in inner
2025-09-07T08:31:33.9052606Z     result_traced = opt_f(*args, **kwargs)
2025-09-07T08:31:33.9052941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 414, in __call__
2025-09-07T08:31:33.9053288Z     return super().__call__(*args, **kwargs)
2025-09-07T08:31:33.9053728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
2025-09-07T08:31:33.9054109Z     return self._call_impl(*args, **kwargs)
2025-09-07T08:31:33.9054446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:33.9054797Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:33.9055154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 841, in compile_wrapper
2025-09-07T08:31:33.9055570Z     raise e.with_traceback(None) from e.__cause__  # User compiler error
2025-09-07T08:31:33.9056146Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not guard on data-dependent expression u0 + 2 > 512 (unhinted: u0 + 2 > 512).  (Size-like symbols: none)
2025-09-07T08:31:33.9056574Z 
2025-09-07T08:31:33.9057195Z consider using data-dependent friendly APIs such as guard_or_false, guard_or_true and statically_known_trueCaused by: if max_word_piece_length + 2 > self._max_position_embeddings:  # fastNLP/embeddings/bert_embedding.py:446 in forward (_dynamo/variables/tensor.py:1435 in evaluate_expr)
2025-09-07T08:31:33.9057996Z For more information, run with TORCH_LOGS="dynamic"
2025-09-07T08:31:33.9058340Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0"
2025-09-07T08:31:33.9058748Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:31:33.9059280Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing
2025-09-07T08:31:33.9059624Z 
2025-09-07T08:31:33.9059704Z User Stack (most recent call last):
2025-09-07T08:31:33.9059903Z   (snipped, see stack below for prefix)
2025-09-07T08:31:33.9060249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T08:31:33.9060584Z     sequence_output = self.bert(words)
2025-09-07T08:31:33.9060939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:33.9061290Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:33.9061662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T08:31:33.9062076Z     outputs = self.model(words)
2025-09-07T08:31:33.9062408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:33.9062750Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:33.9063140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 446, in forward
2025-09-07T08:31:33.9063559Z     if max_word_piece_length + 2 > self._max_position_embeddings:
2025-09-07T08:31:33.9063732Z 
2025-09-07T08:31:33.9063845Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:31:33.9064303Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example
2025-09-07T08:31:33.9064637Z 
2025-09-07T08:31:33.9064701Z from user code:
2025-09-07T08:31:33.9064988Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T08:31:33.9065327Z     sequence_output = self.bert(words)
2025-09-07T08:31:33.9065669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:33.9066022Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:33.9066488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T08:31:33.9066850Z     outputs = self.model(words)
2025-09-07T08:31:33.9067175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:31:33.9067523Z     return forward_call(*args, **kwargs)
2025-09-07T08:31:33.9067913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 446, in forward
2025-09-07T08:31:33.9068319Z     if max_word_piece_length + 2 > self._max_position_embeddings:
2025-09-07T08:31:33.9068500Z 
2025-09-07T08:31:33.9068853Z Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
2025-09-07T08:31:33.9069265Z 
2025-09-07T08:31:33.9069268Z 
2025-09-07T08:31:33.9069766Z The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
2025-09-07T08:31:33.9070454Z TorchDynamo optimized model failed to run because of following error
2025-09-07T08:31:34.0035733Z fail_to_run
2025-09-07T08:31:34.0036083Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:35.3192108Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:31:35.3192978Z   import pynvml  # type: ignore[import]
2025-09-07T08:31:37.3286175Z 
2025-09-07T08:31:37.7620917Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:37.7621227Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:37.7675431Z cpu  eval  functorch_dp_cifar10               
2025-09-07T08:31:37.8337248Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:37.8613411Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:37.8835118Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:42.8462477Z pass
2025-09-07T08:31:42.8462881Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:44.0127653Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:31:44.0129038Z   import pynvml  # type: ignore[import]
2025-09-07T08:31:46.0198043Z 
2025-09-07T08:31:46.2146266Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:46.2146549Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:46.2151254Z cpu  eval  functorch_maml_omniglot            
2025-09-07T08:31:46.2248544Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:46.2329143Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:46.2376390Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:50.9351517Z pass
2025-09-07T08:31:50.9351939Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:52.0606863Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:31:52.0608178Z   import pynvml  # type: ignore[import]
2025-09-07T08:31:54.0655443Z 
2025-09-07T08:31:55.7439131Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:31:55.7439406Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:31:55.7487924Z cpu  eval  hf_Albert                          
2025-09-07T08:31:56.6682751Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:56.9089498Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:31:57.1361737Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:03.7538543Z pass
2025-09-07T08:32:03.7538938Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:05.2152065Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:32:05.2153188Z   import pynvml  # type: ignore[import]
2025-09-07T08:32:07.2278241Z 
2025-09-07T08:32:11.3968518Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:32:11.3969105Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:32:11.4292562Z cpu  eval  hf_Bart                            
2025-09-07T08:32:12.7757879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:13.1312428Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:13.4788402Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:20.5306344Z pass
2025-09-07T08:32:20.5306736Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:22.2254171Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:32:22.2255056Z   import pynvml  # type: ignore[import]
2025-09-07T08:32:24.2291516Z 
2025-09-07T08:32:27.3692408Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:32:27.3692691Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:32:27.3925683Z cpu  eval  hf_Bert                            
2025-09-07T08:32:28.4271207Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:28.7161335Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:28.9876461Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:35.4566143Z pass
2025-09-07T08:32:35.4566603Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:37.0854851Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:32:37.0855726Z   import pynvml  # type: ignore[import]
2025-09-07T08:32:39.1048795Z 
2025-09-07T08:32:44.4912177Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:32:44.4912540Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:32:44.5308931Z cpu  eval  hf_Bert_large                      
2025-09-07T08:32:46.7955258Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:47.4461831Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:48.0870442Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:56.5789338Z pass
2025-09-07T08:32:56.5789729Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:32:58.3530738Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:32:58.3531642Z   import pynvml  # type: ignore[import]
2025-09-07T08:33:00.3749219Z 
2025-09-07T08:33:01.7805413Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:33:01.7806725Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:33:01.7807561Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:33:01.7808213Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:33:01.7809509Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:33:01.7810732Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:33:01.7811547Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:33:01.7812181Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:33:03.6655362Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:33:03.6656096Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:33:05.5048273Z 
2025-09-07T08:33:05.5049427Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:33:05.5310013Z cpu  eval  hf_BigBird                         
2025-09-07T08:33:08.1400032Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:11.2312016Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:14.2488311Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:16.1929308Z ERROR:common:
2025-09-07T08:33:16.1929547Z Traceback (most recent call last):
2025-09-07T08:33:16.1929927Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in check_accuracy
2025-09-07T08:33:16.1930311Z     optimized_model_iter_fn = optimize_ctx(
2025-09-07T08:33:16.1930635Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1523, in export
2025-09-07T08:33:16.1930943Z     ep = torch.export.export(
2025-09-07T08:33:16.1931617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 311, in export
2025-09-07T08:33:16.1931968Z     raise e
2025-09-07T08:33:16.1932258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 277, in export
2025-09-07T08:33:16.1932577Z     return _export(
2025-09-07T08:33:16.1932871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:33:16.1933273Z     raise e
2025-09-07T08:33:16.1933553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:33:16.1933872Z     ep = fn(*args, **kwargs)
2025-09-07T08:33:16.1934216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:33:16.1934583Z     return fn(*args, **kwargs)
2025-09-07T08:33:16.1934892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2255, in _export
2025-09-07T08:33:16.1935211Z     ep = _export_for_training(
2025-09-07T08:33:16.1935503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:33:16.1935812Z     raise e
2025-09-07T08:33:16.1936077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:33:16.1936394Z     ep = fn(*args, **kwargs)
2025-09-07T08:33:16.1936711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:33:16.1937053Z     return fn(*args, **kwargs)
2025-09-07T08:33:16.1937393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
2025-09-07T08:33:16.1937841Z     export_artifact = export_func(
2025-09-07T08:33:16.1938180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1415, in _strict_export
2025-09-07T08:33:16.1938530Z     gm_torch_level = _export_to_torch_ir(
2025-09-07T08:33:16.1938892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 812, in _export_to_torch_ir
2025-09-07T08:33:16.1939260Z     gm_torch_level, _ = torch._dynamo.export(
2025-09-07T08:33:16.1939610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 2002, in inner
2025-09-07T08:33:16.1939948Z     result_traced = opt_f(*args, **kwargs)
2025-09-07T08:33:16.1940289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 414, in __call__
2025-09-07T08:33:16.1940634Z     return super().__call__(*args, **kwargs)
2025-09-07T08:33:16.1941112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
2025-09-07T08:33:16.1941492Z     return self._call_impl(*args, **kwargs)
2025-09-07T08:33:16.1941830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1942286Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1942640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 841, in compile_wrapper
2025-09-07T08:33:16.1943057Z     raise e.with_traceback(None) from e.__cause__  # User compiler error
2025-09-07T08:33:16.1943519Z torch._dynamo.exc.UserError: Could not guard on data-dependent expression Eq(u1, u0) (unhinted: Eq(u1, u0)).  (Size-like symbols: u0, u1)
2025-09-07T08:33:16.1943848Z 
2025-09-07T08:33:16.1944465Z consider using data-dependent friendly APIs such as guard_or_false, guard_or_true and statically_known_trueCaused by: rand_attn = np.stack(rand_attn, axis=0)  # transformers/models/big_bird/modeling_big_bird.py:574 in bigbird_block_sparse_attention (_refs/__init__.py:3992 in _check_stack_inputs)
2025-09-07T08:33:16.1945237Z For more information, run with TORCH_LOGS="dynamic"
2025-09-07T08:33:16.1945587Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0,u1"
2025-09-07T08:33:16.1946044Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:33:16.1946587Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing
2025-09-07T08:33:16.1946950Z 
2025-09-07T08:33:16.1947031Z User Stack (most recent call last):
2025-09-07T08:33:16.1947238Z   (snipped, see stack below for prefix)
2025-09-07T08:33:16.1947697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2303, in forward
2025-09-07T08:33:16.1948110Z     outputs = self.bert(
2025-09-07T08:33:16.1948443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1948796Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1949217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1991, in forward
2025-09-07T08:33:16.1949623Z     encoder_outputs = self.encoder(
2025-09-07T08:33:16.1949958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1950301Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1950680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1585, in forward
2025-09-07T08:33:16.1951078Z     layer_outputs = layer_module(
2025-09-07T08:33:16.1951410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1951752Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1952134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1468, in forward
2025-09-07T08:33:16.1952534Z     self_attention_outputs = self.attention(
2025-09-07T08:33:16.1952882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1953227Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1953612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1381, in forward
2025-09-07T08:33:16.1954004Z     self_outputs = self.self(
2025-09-07T08:33:16.1954326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1954674Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1955065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 455, in forward
2025-09-07T08:33:16.1955576Z     context_layer, attention_probs = self.bigbird_block_sparse_attention(
2025-09-07T08:33:16.1956095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 574, in bigbird_block_sparse_attention
2025-09-07T08:33:16.1956591Z     rand_attn = np.stack(rand_attn, axis=0)
2025-09-07T08:33:16.1956729Z 
2025-09-07T08:33:16.1956846Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:33:16.1957305Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example
2025-09-07T08:33:16.1957634Z 
2025-09-07T08:33:16.1957698Z from user code:
2025-09-07T08:33:16.1958069Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2303, in forward
2025-09-07T08:33:16.1958465Z     outputs = self.bert(
2025-09-07T08:33:16.1958789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1959143Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1959534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1991, in forward
2025-09-07T08:33:16.1959960Z     encoder_outputs = self.encoder(
2025-09-07T08:33:16.1960297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1960639Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1961026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1585, in forward
2025-09-07T08:33:16.1961456Z     layer_outputs = layer_module(
2025-09-07T08:33:16.1961781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1962131Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1962517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1468, in forward
2025-09-07T08:33:16.1962926Z     self_attention_outputs = self.attention(
2025-09-07T08:33:16.1963267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1963617Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1964000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1381, in forward
2025-09-07T08:33:16.1964394Z     self_outputs = self.self(
2025-09-07T08:33:16.1964718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:33:16.1965053Z     return forward_call(*args, **kwargs)
2025-09-07T08:33:16.1965440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 455, in forward
2025-09-07T08:33:16.1965905Z     context_layer, attention_probs = self.bigbird_block_sparse_attention(
2025-09-07T08:33:16.1966418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 574, in bigbird_block_sparse_attention
2025-09-07T08:33:16.1966876Z     rand_attn = np.stack(rand_attn, axis=0)
2025-09-07T08:33:16.1967007Z 
2025-09-07T08:33:16.1967362Z Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
2025-09-07T08:33:16.1967771Z 
2025-09-07T08:33:16.1967773Z 
2025-09-07T08:33:16.1968271Z The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
2025-09-07T08:33:16.1969013Z TorchDynamo optimized model failed to run because of following error
2025-09-07T08:33:16.3898457Z fail_to_run
2025-09-07T08:33:16.3898803Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:18.1733207Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:33:18.1734436Z   import pynvml  # type: ignore[import]
2025-09-07T08:33:20.1779176Z 
2025-09-07T08:33:22.7969366Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:33:22.7970024Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:33:22.8104910Z cpu  eval  hf_DistilBert                      
2025-09-07T08:33:23.3980680Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:23.5624760Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:23.7161624Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:29.0128842Z pass
2025-09-07T08:33:29.0129551Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:30.5699476Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:33:30.5700363Z   import pynvml  # type: ignore[import]
2025-09-07T08:33:32.5892259Z 
2025-09-07T08:33:37.2496321Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:33:37.2496659Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:33:37.2674826Z cpu  eval  hf_GPT2                            
2025-09-07T08:33:37.8203232Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:37.9820095Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:38.1362671Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:45.6604116Z pass
2025-09-07T08:33:45.6604515Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:33:47.3320872Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:33:47.3321767Z   import pynvml  # type: ignore[import]
2025-09-07T08:33:49.3516252Z 
2025-09-07T08:34:03.6054261Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:34:03.6054603Z loading model: 0it [00:14, ?it/s]
2025-09-07T08:34:03.6666779Z cpu  eval  hf_GPT2_large                      
2025-09-07T08:34:04.0591428Z pass_due_to_skip
2025-09-07T08:34:04.0595784Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:06.4948548Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:34:06.4949402Z   import pynvml  # type: ignore[import]
2025-09-07T08:34:08.5082966Z 
2025-09-07T08:34:12.5491440Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:34:12.5491805Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:34:12.5783022Z cpu  eval  hf_Longformer                      
2025-09-07T08:34:14.2844242Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:15.0234530Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:15.8091895Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:16.9129326Z ERROR:common:
2025-09-07T08:34:16.9129937Z Traceback (most recent call last):
2025-09-07T08:34:16.9130316Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in check_accuracy
2025-09-07T08:34:16.9130695Z     optimized_model_iter_fn = optimize_ctx(
2025-09-07T08:34:16.9131022Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1523, in export
2025-09-07T08:34:16.9131340Z     ep = torch.export.export(
2025-09-07T08:34:16.9131708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 311, in export
2025-09-07T08:34:16.9132069Z     raise e
2025-09-07T08:34:16.9132372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 277, in export
2025-09-07T08:34:16.9132693Z     return _export(
2025-09-07T08:34:16.9132987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:34:16.9133299Z     raise e
2025-09-07T08:34:16.9133663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:34:16.9133995Z     ep = fn(*args, **kwargs)
2025-09-07T08:34:16.9134345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:34:16.9134702Z     return fn(*args, **kwargs)
2025-09-07T08:34:16.9135108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2255, in _export
2025-09-07T08:34:16.9135434Z     ep = _export_for_training(
2025-09-07T08:34:16.9135739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:34:16.9136053Z     raise e
2025-09-07T08:34:16.9136324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:34:16.9136640Z     ep = fn(*args, **kwargs)
2025-09-07T08:34:16.9136958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:34:16.9137308Z     return fn(*args, **kwargs)
2025-09-07T08:34:16.9137704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
2025-09-07T08:34:16.9138068Z     export_artifact = export_func(
2025-09-07T08:34:16.9138401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1415, in _strict_export
2025-09-07T08:34:16.9138758Z     gm_torch_level = _export_to_torch_ir(
2025-09-07T08:34:16.9139116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 812, in _export_to_torch_ir
2025-09-07T08:34:16.9139488Z     gm_torch_level, _ = torch._dynamo.export(
2025-09-07T08:34:16.9139837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 2002, in inner
2025-09-07T08:34:16.9140173Z     result_traced = opt_f(*args, **kwargs)
2025-09-07T08:34:16.9140515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 414, in __call__
2025-09-07T08:34:16.9140862Z     return super().__call__(*args, **kwargs)
2025-09-07T08:34:16.9141242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
2025-09-07T08:34:16.9141615Z     return self._call_impl(*args, **kwargs)
2025-09-07T08:34:16.9141958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9142305Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9142659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 841, in compile_wrapper
2025-09-07T08:34:16.9143151Z     raise e.with_traceback(None) from e.__cause__  # User compiler error
2025-09-07T08:34:16.9143715Z torch._dynamo.exc.UserError: Consider annotating your code using torch._check*(). Could not guard on data-dependent expression Eq(u0, 1) (unhinted: Eq(u0, 1)).  (Size-like symbols: none)
2025-09-07T08:34:16.9144186Z 
2025-09-07T08:34:16.9144745Z consider using data-dependent friendly APIs such as guard_or_false, guard_or_true and statically_known_trueCaused by: if is_global_attn:  # transformers/models/longformer/modeling_longformer.py:554 in forward (_dynamo/variables/tensor.py:1435 in evaluate_expr)
2025-09-07T08:34:16.9145475Z For more information, run with TORCH_LOGS="dynamic"
2025-09-07T08:34:16.9145817Z For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0"
2025-09-07T08:34:16.9146228Z If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:34:16.9146759Z For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing
2025-09-07T08:34:16.9147102Z 
2025-09-07T08:34:16.9147176Z User Stack (most recent call last):
2025-09-07T08:34:16.9147379Z   (snipped, see stack below for prefix)
2025-09-07T08:34:16.9147856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T08:34:16.9148293Z     outputs = self.longformer(
2025-09-07T08:34:16.9148637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9148988Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9149458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T08:34:16.9149878Z     encoder_outputs = self.encoder(
2025-09-07T08:34:16.9150224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9150582Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9150994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in forward
2025-09-07T08:34:16.9151412Z     layer_outputs = layer_module(
2025-09-07T08:34:16.9151749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9152093Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9152504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:34:16.9152916Z     self_attn_outputs = self.attention(
2025-09-07T08:34:16.9153256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9153610Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9154011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:34:16.9154417Z     self_outputs = self.self(
2025-09-07T08:34:16.9154748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9155097Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9155511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 554, in forward
2025-09-07T08:34:16.9155917Z     if is_global_attn:
2025-09-07T08:34:16.9156015Z 
2025-09-07T08:34:16.9156126Z For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
2025-09-07T08:34:16.9156586Z For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example
2025-09-07T08:34:16.9156927Z 
2025-09-07T08:34:16.9157028Z from user code:
2025-09-07T08:34:16.9157415Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T08:34:16.9157831Z     outputs = self.longformer(
2025-09-07T08:34:16.9158154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9158540Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9158953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T08:34:16.9159373Z     encoder_outputs = self.encoder(
2025-09-07T08:34:16.9159718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9160071Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9160485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in forward
2025-09-07T08:34:16.9160906Z     layer_outputs = layer_module(
2025-09-07T08:34:16.9161239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9161576Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9162020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T08:34:16.9162443Z     self_attn_outputs = self.attention(
2025-09-07T08:34:16.9162788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9163130Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9163559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T08:34:16.9163975Z     self_outputs = self.self(
2025-09-07T08:34:16.9164299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:16.9164641Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:16.9165043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 554, in forward
2025-09-07T08:34:16.9165448Z     if is_global_attn:
2025-09-07T08:34:16.9165551Z 
2025-09-07T08:34:16.9165901Z Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
2025-09-07T08:34:16.9166308Z 
2025-09-07T08:34:16.9166311Z 
2025-09-07T08:34:16.9166812Z The error above occurred when calling torch.export.export. If you would like to view some more information about this error, and get a list of all other errors that may occur in your export call, you can replace your `export()` call with `draft_export()`.
2025-09-07T08:34:16.9167496Z TorchDynamo optimized model failed to run because of following error
2025-09-07T08:34:17.0739065Z fail_to_run
2025-09-07T08:34:17.0739424Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:18.7522610Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:34:18.7523499Z   import pynvml  # type: ignore[import]
2025-09-07T08:34:20.7668288Z 
2025-09-07T08:34:22.1438843Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:34:22.1440589Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:34:22.9080436Z 
2025-09-07T08:34:22.9080942Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:34:22.9127213Z cpu  eval  hf_Reformer                        
2025-09-07T08:34:23.4240126Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:24.9017676Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:26.3721818Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:27.1854901Z class GraphModule(torch.nn.Module):
2025-09-07T08:34:27.1855231Z     def forward(self, L_input_ids_: "i64[4, 2048][2048, 1]cpu"):
2025-09-07T08:34:27.1855478Z         l_input_ids_ = L_input_ids_
2025-09-07T08:34:27.1855672Z         
2025-09-07T08:34:27.1856320Z          # File: /opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py:194 in _get_least_common_mult_chunk_len, code: return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length)
2025-09-07T08:34:27.1857469Z         least_common_mult_chunk_length: "i64[][]cpu" = torch__dynamo_utils_wrapped_lcm(64, 64)
2025-09-07T08:34:27.1857816Z         
2025-09-07T08:34:27.1858310Z          # File: /opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py:2099 in forward, code: input_shape[-1] % least_common_mult_chunk_length != 0
2025-09-07T08:34:27.1859136Z         wrapped_mod: "i64[][]cpu" = torch__dynamo_utils_wrapped_mod(2048, least_common_mult_chunk_length);  least_common_mult_chunk_length = None
2025-09-07T08:34:27.1859651Z         wrapped_ne: "b8[][]cpu" = torch__dynamo_utils_wrapped_ne(wrapped_mod, 0);  wrapped_mod = wrapped_ne = None
2025-09-07T08:34:27.1859968Z         
2025-09-07T08:34:27.1860122Z class GraphModule(torch.nn.Module):
2025-09-07T08:34:27.1860364Z     def forward(self, L_input_ids_: "i64[4, 2048][2048, 1]cpu"):
2025-09-07T08:34:27.1860592Z         l_input_ids_ = L_input_ids_
2025-09-07T08:34:27.1860767Z         
2025-09-07T08:34:27.1861375Z          # File: /opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py:194 in _get_least_common_mult_chunk_len, code: return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length)
2025-09-07T08:34:27.1862089Z         least_common_mult_chunk_length: "i64[][]cpu" = torch__dynamo_utils_wrapped_lcm(64, 64)
2025-09-07T08:34:27.1862366Z         
2025-09-07T08:34:27.1862838Z          # File: /opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py:2099 in forward, code: input_shape[-1] % least_common_mult_chunk_length != 0
2025-09-07T08:34:27.1863548Z         wrapped_mod: "i64[][]cpu" = torch__dynamo_utils_wrapped_mod(2048, least_common_mult_chunk_length);  least_common_mult_chunk_length = None
2025-09-07T08:34:27.1864055Z         wrapped_ne: "b8[][]cpu" = torch__dynamo_utils_wrapped_ne(wrapped_mod, 0);  wrapped_mod = wrapped_ne = None
2025-09-07T08:34:27.1864355Z         
2025-09-07T08:34:27.1864485Z ERROR:common:
2025-09-07T08:34:27.1864635Z Traceback (most recent call last):
2025-09-07T08:34:27.1864966Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2320, in check_accuracy
2025-09-07T08:34:27.1865300Z     optimized_model_iter_fn = optimize_ctx(
2025-09-07T08:34:27.1865612Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1523, in export
2025-09-07T08:34:27.1865910Z     ep = torch.export.export(
2025-09-07T08:34:27.1866250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 311, in export
2025-09-07T08:34:27.1866569Z     raise e
2025-09-07T08:34:27.1866844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/__init__.py", line 277, in export
2025-09-07T08:34:27.1867258Z     return _export(
2025-09-07T08:34:27.1867550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:34:27.1867868Z     raise e
2025-09-07T08:34:27.1868140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:34:27.1868557Z     ep = fn(*args, **kwargs)
2025-09-07T08:34:27.1868897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:34:27.1869245Z     return fn(*args, **kwargs)
2025-09-07T08:34:27.1869552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2255, in _export
2025-09-07T08:34:27.1869876Z     ep = _export_for_training(
2025-09-07T08:34:27.1870178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1163, in wrapper
2025-09-07T08:34:27.1870484Z     raise e
2025-09-07T08:34:27.1870758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1129, in wrapper
2025-09-07T08:34:27.1871075Z     ep = fn(*args, **kwargs)
2025-09-07T08:34:27.1871400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/exported_program.py", line 124, in wrapper
2025-09-07T08:34:27.1871755Z     return fn(*args, **kwargs)
2025-09-07T08:34:27.1872127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 2071, in _export_for_training
2025-09-07T08:34:27.1872501Z     export_artifact = export_func(
2025-09-07T08:34:27.1872856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 1415, in _strict_export
2025-09-07T08:34:27.1873216Z     gm_torch_level = _export_to_torch_ir(
2025-09-07T08:34:27.1873605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/export/_trace.py", line 812, in _export_to_torch_ir
2025-09-07T08:34:27.1873983Z     gm_torch_level, _ = torch._dynamo.export(
2025-09-07T08:34:27.1874329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 2002, in inner
2025-09-07T08:34:27.1874680Z     result_traced = opt_f(*args, **kwargs)
2025-09-07T08:34:27.1875024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 414, in __call__
2025-09-07T08:34:27.1875376Z     return super().__call__(*args, **kwargs)
2025-09-07T08:34:27.1875757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
2025-09-07T08:34:27.1876137Z     return self._call_impl(*args, **kwargs)
2025-09-07T08:34:27.1876491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:27.1876841Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:27.1877206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 841, in compile_wrapper
2025-09-07T08:34:27.1877620Z     raise e.with_traceback(None) from e.__cause__  # User compiler error
2025-09-07T08:34:27.1877923Z torch._dynamo.exc.Unsupported: Data-dependent branching
2025-09-07T08:34:27.1878357Z   Explanation: Detected data-dependent branching (e.g. `if my_tensor.sum() > 0:`). Dynamo does not support tracing dynamic control flow.
2025-09-07T08:34:27.1878945Z   Hint: This graph break is fundamental - it is unlikely that Dynamo will ever be able to trace through your code. Consider finding a workaround.
2025-09-07T08:34:27.1879390Z   Hint: Use `torch.cond` to express dynamic control flow.
2025-09-07T08:34:27.1879556Z 
2025-09-07T08:34:27.1879695Z   Developer debug context: attempted to jump with NumpyNdarrayVariable()
2025-09-07T08:34:27.1879894Z 
2025-09-07T08:34:27.1880184Z  For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0170.html
2025-09-07T08:34:27.1880503Z 
2025-09-07T08:34:27.1880568Z from user code:
2025-09-07T08:34:27.1880992Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2481, in forward
2025-09-07T08:34:27.1881416Z     reformer_outputs = self.reformer(
2025-09-07T08:34:27.1881766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
2025-09-07T08:34:27.1882167Z     return forward_call(*args, **kwargs)
2025-09-07T08:34:27.1882573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2099, in forward
2025-09-07T08:34:27.1883024Z     input_shape[-1] % least_common_mult_chunk_length != 0
2025-09-07T08:34:27.1883188Z 
2025-09-07T08:34:27.1883536Z Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
2025-09-07T08:34:27.1883944Z 
2025-09-07T08:34:27.1884080Z TorchDynamo optimized model failed to run because of following error
2025-09-07T08:34:27.2618279Z fail_to_run
2025-09-07T08:34:27.2618622Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:28.5499135Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:34:28.5500044Z   import pynvml  # type: ignore[import]
2025-09-07T08:34:30.5738501Z 
2025-09-07T08:34:36.3887173Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:34:36.3887652Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:34:36.4218471Z cpu  eval  hf_Roberta_base                    
2025-09-07T08:34:38.3125015Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:38.8458510Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:39.3812901Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:47.7760955Z pass
2025-09-07T08:34:47.7761381Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:50.1234167Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:34:50.1235043Z   import pynvml  # type: ignore[import]
2025-09-07T08:34:52.1410528Z 
2025-09-07T08:34:55.0227380Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:34:55.0227729Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:34:55.0447517Z cpu  eval  hf_T5                              
2025-09-07T08:34:56.0469830Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:56.5111562Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:34:56.9612658Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:35:04.7318957Z pass
2025-09-07T08:35:04.7319396Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:35:06.4315574Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:35:06.4316475Z   import pynvml  # type: ignore[import]
2025-09-07T08:35:08.4407438Z 
2025-09-07T08:35:17.8935671Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:35:17.8936007Z loading model: 0it [00:09, ?it/s]
2025-09-07T08:35:17.9339282Z cpu  eval  hf_T5_base                         
2025-09-07T08:35:32.3534128Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:35:38.8444558Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:35:45.2895648Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:36:11.6535767Z pass
2025-09-07T08:36:11.6536167Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:36:16.9724855Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:36:16.9725959Z   import pynvml  # type: ignore[import]
2025-09-07T08:36:18.9850592Z 
2025-09-07T08:36:31.5177764Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:36:31.5178325Z loading model: 0it [00:12, ?it/s]
2025-09-07T08:36:31.6025014Z cpu  eval  hf_T5_large                        
2025-09-07T08:36:32.0029131Z pass_due_to_skip
2025-09-07T08:36:32.0032770Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:36:34.3331375Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:36:34.3332258Z   import pynvml  # type: ignore[import]
2025-09-07T08:36:36.3476945Z 
2025-09-07T08:36:42.5975094Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:36:42.5975399Z loading model: 0it [00:06, ?it/s]
2025-09-07T08:36:42.6358718Z cpu  eval  hf_distil_whisper                  
2025-09-07T08:36:50.0496551Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:36:52.1460243Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:36:54.2316303Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:04.7195052Z pass
2025-09-07T08:37:04.7195467Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:06.6759844Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:37:06.6760717Z   import pynvml  # type: ignore[import]
2025-09-07T08:37:08.6983645Z 
2025-09-07T08:37:08.8523134Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:37:08.8523431Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:37:08.8523655Z cpu  eval  lennard_jones                      
2025-09-07T08:37:08.8534986Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:08.8560465Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:08.8575307Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:13.5120166Z pass
2025-09-07T08:37:13.5120574Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:14.2201053Z accuracy            pass_rate=65.22%
2025-09-07T08:37:14.2205102Z calls_captured      gmean=0.00x mean=263.391x
2025-09-07T08:37:14.2207477Z unique_graphs       gmean=0.00x mean=0.652x
2025-09-07T08:37:14.2209564Z graph_breaks        gmean=0.00x mean=0.000x
2025-09-07T08:37:14.2211690Z unique_graph_breaks gmean=0.00x mean=0.000x
2025-09-07T08:37:14.2213877Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T08:37:14.2216052Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T08:37:14.2217962Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T08:37:14.2218662Z compilation_latency mean=4.096 seconds
2025-09-07T08:37:14.7479623Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv
2025-09-07T08:37:15.1433620Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:37:15.1434475Z   import pynvml  # type: ignore[import]
2025-09-07T08:37:17.5924593Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:37:17.5926127Z   import pynvml  # type: ignore[import]
2025-09-07T08:37:19.6130870Z 
2025-09-07T08:37:24.8780882Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:37:24.8781237Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:37:24.9059463Z cpu  eval  dlrm                               
2025-09-07T08:37:25.0430125Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:25.0712867Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:25.0970766Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:44.6408023Z pass
2025-09-07T08:37:44.6410583Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:46.5723338Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:37:46.5724230Z   import pynvml  # type: ignore[import]
2025-09-07T08:37:48.5861028Z 
2025-09-07T08:37:49.0211412Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:37:49.0211692Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:37:49.0243278Z cpu  eval  functorch_dp_cifar10               
2025-09-07T08:37:49.0913700Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:49.1185198Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:49.1412373Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:37:59.6697842Z pass
2025-09-07T08:37:59.6700190Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:01.4273749Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:38:01.4274647Z   import pynvml  # type: ignore[import]
2025-09-07T08:38:03.4458559Z 
2025-09-07T08:38:03.6402649Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:38:03.6403314Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:38:03.6408132Z cpu  eval  functorch_maml_omniglot            
2025-09-07T08:38:03.6503338Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:03.6581027Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:03.6629217Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:11.5172422Z pass
2025-09-07T08:38:11.5174714Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:13.0291111Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:38:13.0292006Z   import pynvml  # type: ignore[import]
2025-09-07T08:38:15.0278916Z 
2025-09-07T08:38:16.6958340Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:38:16.6958835Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:38:16.7006164Z cpu  eval  hf_Albert                          
2025-09-07T08:38:17.6140975Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:17.8557987Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:18.0830639Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:30.0639391Z pass
2025-09-07T08:38:30.0639800Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:32.1960880Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:38:32.1961766Z   import pynvml  # type: ignore[import]
2025-09-07T08:38:34.2104656Z 
2025-09-07T08:38:38.4429828Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:38:38.4430794Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:38:38.4690774Z cpu  eval  hf_Bart                            
2025-09-07T08:38:39.8095924Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:40.1578313Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:40.5060568Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:56.9680090Z pass
2025-09-07T08:38:56.9680476Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:38:59.5200055Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:38:59.5200933Z   import pynvml  # type: ignore[import]
2025-09-07T08:39:01.5420460Z 
2025-09-07T08:39:04.6738714Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:39:04.6739040Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:39:04.6973428Z cpu  eval  hf_Bert                            
2025-09-07T08:39:05.7895980Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:06.0766254Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:06.3523517Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:20.0822608Z pass
2025-09-07T08:39:20.0823040Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:22.3920171Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:39:22.3921406Z   import pynvml  # type: ignore[import]
2025-09-07T08:39:24.4025691Z 
2025-09-07T08:39:29.8114124Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:39:29.8114848Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:39:29.8500087Z cpu  eval  hf_Bert_large                      
2025-09-07T08:39:32.1164637Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:32.7673704Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:33.4072239Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:53.9832572Z pass
2025-09-07T08:39:53.9832977Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:39:56.7302105Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:39:56.7303021Z   import pynvml  # type: ignore[import]
2025-09-07T08:39:58.7465171Z 
2025-09-07T08:40:00.1488689Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:40:00.1490355Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:40:00.1491195Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:40:00.1491848Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:40:00.1493151Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:40:00.1494361Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:40:00.1495173Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:40:00.1495805Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:40:01.9783179Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:40:01.9783965Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:40:03.8081202Z 
2025-09-07T08:40:03.8082237Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:40:03.8373657Z cpu  eval  hf_BigBird                         
2025-09-07T08:40:06.3139571Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:40:09.3269645Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:40:12.4237063Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:17.3992174Z pass
2025-09-07T08:41:17.3992583Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:21.4516938Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:41:21.4518040Z   import pynvml  # type: ignore[import]
2025-09-07T08:41:23.4740724Z 
2025-09-07T08:41:26.0897512Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:41:26.0897807Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:41:26.1007696Z cpu  eval  hf_DistilBert                      
2025-09-07T08:41:26.6822473Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:26.8458267Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:27.0005720Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:38.1747812Z pass
2025-09-07T08:41:38.1748556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:40.3640473Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:41:40.3641712Z   import pynvml  # type: ignore[import]
2025-09-07T08:41:42.3785153Z 
2025-09-07T08:41:47.0203374Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:41:47.0203707Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:41:47.0382531Z cpu  eval  hf_GPT2                            
2025-09-07T08:41:47.5880209Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:47.7514752Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:41:47.9073039Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:02.2466757Z pass
2025-09-07T08:42:02.2467162Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:04.6397283Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:42:04.6398177Z   import pynvml  # type: ignore[import]
2025-09-07T08:42:06.6495481Z 
2025-09-07T08:42:20.8970973Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:42:20.8971474Z loading model: 0it [00:14, ?it/s]
2025-09-07T08:42:20.9576290Z cpu  eval  hf_GPT2_large                      
2025-09-07T08:42:21.3455799Z pass_due_to_skip
2025-09-07T08:42:21.3459990Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:23.8005142Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:42:23.8006036Z   import pynvml  # type: ignore[import]
2025-09-07T08:42:25.8119337Z 
2025-09-07T08:42:31.5946531Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:42:31.5947266Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:42:31.6231513Z cpu  eval  hf_Roberta_base                    
2025-09-07T08:42:33.5253074Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:34.0613892Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:34.5985464Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:53.9655108Z pass
2025-09-07T08:42:53.9655495Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:42:57.2914229Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:42:57.2915119Z   import pynvml  # type: ignore[import]
2025-09-07T08:42:59.2980620Z 
2025-09-07T08:43:02.1408739Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:43:02.1409037Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:43:02.1620730Z cpu  eval  hf_T5                              
2025-09-07T08:43:03.1440566Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:43:03.6116623Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:43:04.0672554Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:43:21.0515034Z pass
2025-09-07T08:43:21.0519878Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:43:23.5164239Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:43:23.5165250Z   import pynvml  # type: ignore[import]
2025-09-07T08:43:25.5263168Z 
2025-09-07T08:43:35.1524535Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:43:35.1524856Z loading model: 0it [00:09, ?it/s]
2025-09-07T08:43:35.1940687Z cpu  eval  hf_T5_base                         
2025-09-07T08:43:49.5763628Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:43:56.0758067Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:44:02.5616491Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:44:50.4827110Z pass
2025-09-07T08:44:50.4827540Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:44:56.9025179Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:44:56.9026112Z   import pynvml  # type: ignore[import]
2025-09-07T08:44:58.9137030Z 
2025-09-07T08:45:11.2923345Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:45:11.2923669Z loading model: 0it [00:12, ?it/s]
2025-09-07T08:45:11.3761923Z cpu  eval  hf_T5_large                        
2025-09-07T08:45:11.7720489Z pass_due_to_skip
2025-09-07T08:45:11.7723274Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:45:14.1173521Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:45:14.1174391Z   import pynvml  # type: ignore[import]
2025-09-07T08:45:16.1299197Z 
2025-09-07T08:45:22.3460909Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:45:22.3461184Z loading model: 0it [00:06, ?it/s]
2025-09-07T08:45:22.3845095Z cpu  eval  hf_distil_whisper                  
2025-09-07T08:45:29.7897173Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:45:31.8852909Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:45:34.0392569Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:45:58.2335811Z pass
2025-09-07T08:45:58.2336214Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:46:01.1367762Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:46:01.1368733Z   import pynvml  # type: ignore[import]
2025-09-07T08:46:03.1393524Z 
2025-09-07T08:46:03.2922704Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:46:03.2922976Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:46:03.2924365Z cpu  eval  lennard_jones                      
2025-09-07T08:46:03.2937498Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:46:03.2963583Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:46:03.2978501Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:46:10.2483856Z pass
2025-09-07T08:46:10.2486530Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:46:11.3323273Z accuracy            pass_rate=88.24%
2025-09-07T08:46:11.3326216Z calls_captured      gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3328596Z unique_graphs       gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3330727Z graph_breaks        gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3332837Z unique_graph_breaks gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3334961Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3337172Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3339384Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T08:46:11.3340416Z compilation_latency mean=0.000 seconds
2025-09-07T08:46:11.8628645Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *maxautotune-true* ]]
2025-09-07T08:46:11.8629454Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *cudagraphs_low_precision-true* ]]
2025-09-07T08:46:11.8629962Z + for target in "${targets[@]}"
2025-09-07T08:46:11.8630159Z + target_flag=('--performance')
2025-09-07T08:46:11.8630339Z + local target_flag
2025-09-07T08:46:11.8630521Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]]
2025-09-07T08:46:11.8630734Z + target_flag+=(--cold-start-latency)
2025-09-07T08:46:11.8631188Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *freezing-true* ]]
2025-09-07T08:46:11.8631860Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *default-true* ]]
2025-09-07T08:46:11.8633040Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv
2025-09-07T08:46:12.2614513Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:46:12.2615897Z   import pynvml  # type: ignore[import]
2025-09-07T08:46:14.7009183Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:46:14.7010386Z   import pynvml  # type: ignore[import]
2025-09-07T08:46:16.7219066Z 
2025-09-07T08:46:22.3804991Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:46:22.3805598Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:46:22.4012045Z cpu  eval  dlrm                               
2025-09-07T08:46:35.3560380Z 
2025-09-07T08:46:35.4917561Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:46:35.6250608Z running benchmark:   7% 2/30 [00:00<00:01, 14.78it/s]
2025-09-07T08:46:35.7570870Z running benchmark:  13% 4/30 [00:00<00:01, 14.91it/s]
2025-09-07T08:46:35.8890666Z running benchmark:  20% 6/30 [00:00<00:01, 15.01it/s]
2025-09-07T08:46:36.0210403Z running benchmark:  27% 8/30 [00:00<00:01, 15.07it/s]
2025-09-07T08:46:36.1530188Z running benchmark:  33% 10/30 [00:00<00:01, 15.10it/s]
2025-09-07T08:46:36.2849671Z running benchmark:  40% 12/30 [00:00<00:01, 15.12it/s]
2025-09-07T08:46:36.4169719Z running benchmark:  47% 14/30 [00:00<00:01, 15.13it/s]
2025-09-07T08:46:36.5492240Z running benchmark:  53% 16/30 [00:01<00:00, 15.14it/s]
2025-09-07T08:46:36.6820733Z running benchmark:  60% 18/30 [00:01<00:00, 15.13it/s]
2025-09-07T08:46:36.8142718Z running benchmark:  67% 20/30 [00:01<00:00, 15.11it/s]
2025-09-07T08:46:36.9472227Z running benchmark:  73% 22/30 [00:01<00:00, 15.11it/s]
2025-09-07T08:46:37.0788674Z running benchmark:  80% 24/30 [00:01<00:00, 15.09it/s]
2025-09-07T08:46:37.2111503Z running benchmark:  87% 26/30 [00:01<00:00, 15.12it/s]
2025-09-07T08:46:37.3439902Z running benchmark:  93% 28/30 [00:01<00:00, 15.12it/s]
2025-09-07T08:46:37.3440229Z running benchmark: 100% 30/30 [00:01<00:00, 15.10it/s]
2025-09-07T08:46:37.3440502Z running benchmark: 100% 30/30 [00:01<00:00, 15.09it/s]
2025-09-07T08:46:37.4446659Z 1.012x
2025-09-07T08:46:37.4449791Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:46:39.5756176Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:46:39.5757040Z   import pynvml  # type: ignore[import]
2025-09-07T08:46:41.5915520Z 
2025-09-07T08:46:44.2260897Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:46:44.4641362Z 
2025-09-07T08:46:44.4641754Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:46:44.4642183Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:46:44.4642537Z cpu  eval  doctr_det_predictor                
2025-09-07T08:46:44.4642866Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T08:47:30.7532663Z 
2025-09-07T08:47:30.9585536Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:47:31.1526423Z running benchmark:   3% 1/30 [00:00<00:05,  4.88it/s]
2025-09-07T08:47:31.3434812Z running benchmark:   7% 2/30 [00:00<00:05,  5.03it/s]
2025-09-07T08:47:31.5355892Z running benchmark:  10% 3/30 [00:00<00:05,  5.13it/s]
2025-09-07T08:47:31.7979551Z running benchmark:  13% 4/30 [00:00<00:05,  5.16it/s]
2025-09-07T08:47:31.9898993Z running benchmark:  17% 5/30 [00:01<00:05,  4.57it/s]
2025-09-07T08:47:32.1821143Z running benchmark:  20% 6/30 [00:01<00:05,  4.77it/s]
2025-09-07T08:47:32.3728345Z running benchmark:  23% 7/30 [00:01<00:04,  4.90it/s]
2025-09-07T08:47:32.5655284Z running benchmark:  27% 8/30 [00:01<00:04,  5.01it/s]
2025-09-07T08:47:32.7578308Z running benchmark:  30% 9/30 [00:01<00:04,  5.06it/s]
2025-09-07T08:47:32.9498678Z running benchmark:  33% 10/30 [00:02<00:03,  5.10it/s]
2025-09-07T08:47:33.1425959Z running benchmark:  37% 11/30 [00:02<00:03,  5.14it/s]
2025-09-07T08:47:33.3355798Z running benchmark:  40% 12/30 [00:02<00:03,  5.15it/s]
2025-09-07T08:47:33.5280584Z running benchmark:  43% 13/30 [00:02<00:03,  5.16it/s]
2025-09-07T08:47:33.7211509Z running benchmark:  47% 14/30 [00:02<00:03,  5.17it/s]
2025-09-07T08:47:33.9143048Z running benchmark:  50% 15/30 [00:02<00:02,  5.17it/s]
2025-09-07T08:47:34.1045965Z running benchmark:  53% 16/30 [00:03<00:02,  5.17it/s]
2025-09-07T08:47:34.2970591Z running benchmark:  57% 17/30 [00:03<00:02,  5.20it/s]
2025-09-07T08:47:34.4900871Z running benchmark:  60% 18/30 [00:03<00:02,  5.20it/s]
2025-09-07T08:47:34.6821559Z running benchmark:  63% 19/30 [00:03<00:02,  5.19it/s]
2025-09-07T08:47:34.8744296Z running benchmark:  67% 20/30 [00:03<00:01,  5.20it/s]
2025-09-07T08:47:35.0674396Z running benchmark:  70% 21/30 [00:04<00:01,  5.20it/s]
2025-09-07T08:47:35.2591654Z running benchmark:  73% 22/30 [00:04<00:01,  5.19it/s]
2025-09-07T08:47:35.4515586Z running benchmark:  77% 23/30 [00:04<00:01,  5.20it/s]
2025-09-07T08:47:35.6432339Z running benchmark:  80% 24/30 [00:04<00:01,  5.20it/s]
2025-09-07T08:47:35.9032595Z running benchmark:  83% 25/30 [00:04<00:00,  5.21it/s]
2025-09-07T08:47:36.0958651Z running benchmark:  87% 26/30 [00:05<00:00,  4.71it/s]
2025-09-07T08:47:36.2870659Z running benchmark:  90% 27/30 [00:05<00:00,  4.84it/s]
2025-09-07T08:47:36.4808292Z running benchmark:  93% 28/30 [00:05<00:00,  4.95it/s]
2025-09-07T08:47:36.6737213Z running benchmark:  97% 29/30 [00:05<00:00,  5.01it/s]
2025-09-07T08:47:36.6738054Z running benchmark: 100% 30/30 [00:05<00:00,  5.06it/s]
2025-09-07T08:47:36.6738350Z running benchmark: 100% 30/30 [00:05<00:00,  5.07it/s]
2025-09-07T08:47:36.7979940Z 2.151x
2025-09-07T08:47:36.7983644Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:47:39.3581077Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:47:39.3581984Z   import pynvml  # type: ignore[import]
2025-09-07T08:47:41.3648208Z 
2025-09-07T08:47:43.9553539Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:47:43.9890489Z 
2025-09-07T08:47:43.9890848Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:47:43.9891312Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:47:43.9891704Z cpu  eval  doctr_reco_predictor               
2025-09-07T08:47:43.9892196Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T08:48:01.1872785Z 
2025-09-07T08:48:01.3037320Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:48:01.4143888Z running benchmark:  20% 6/30 [00:00<00:00, 51.71it/s]
2025-09-07T08:48:01.5242920Z running benchmark:  40% 12/30 [00:00<00:00, 53.13it/s]
2025-09-07T08:48:01.6349525Z running benchmark:  60% 18/30 [00:00<00:00, 53.80it/s]
2025-09-07T08:48:01.7453471Z running benchmark:  80% 24/30 [00:00<00:00, 53.96it/s]
2025-09-07T08:48:01.7453810Z running benchmark: 100% 30/30 [00:00<00:00, 54.11it/s]
2025-09-07T08:48:01.7454077Z running benchmark: 100% 30/30 [00:00<00:00, 53.78it/s]
2025-09-07T08:48:01.8184156Z 1.433x
2025-09-07T08:48:01.8187948Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:48:04.0317327Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:48:04.0318523Z   import pynvml  # type: ignore[import]
2025-09-07T08:48:06.0435276Z 
2025-09-07T08:48:06.0964783Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T08:48:06.0965769Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T08:48:06.0966378Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T08:48:06.0966964Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T08:48:06.9411439Z 
2025-09-07T08:48:06.9411855Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:48:06.9424116Z cpu  eval  drq                                
2025-09-07T08:48:19.2471806Z 
2025-09-07T08:48:19.3488507Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:48:19.4340981Z running benchmark:  53% 16/30 [00:00<00:00, 158.00it/s]
2025-09-07T08:48:19.4341310Z running benchmark: 100% 30/30 [00:00<00:00, 160.81it/s]
2025-09-07T08:48:19.4878888Z 0.754x
2025-09-07T08:48:19.4883002Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:48:21.5425046Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:48:21.5425982Z   import pynvml  # type: ignore[import]
2025-09-07T08:48:23.5621625Z 
2025-09-07T08:48:24.7160764Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:48:24.7161096Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:48:24.7322406Z cpu  eval  fastNLP_Bert                       
2025-09-07T08:48:25.4557231Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:48:25.4557917Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:48:25.4558492Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T08:48:25.4558982Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:48:25.4559558Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T08:48:25.4560039Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:48:25.4560506Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T08:48:25.4561193Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T08:48:25.4561869Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(*inputs)
2025-09-07T08:48:25.4562542Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T08:48:25.4563216Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     sequence_output = self.bert(words)
2025-09-07T08:48:25.4563926Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T08:48:25.4564934Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.model(words)
2025-09-07T08:48:25.4565623Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T08:48:25.4567002Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T08:48:25.4570307Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:48:25.4570704Z W0907 08:48:25.454953 132141 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:48:41.8264684Z 
2025-09-07T08:48:41.9421173Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:48:42.0552186Z running benchmark:   3% 1/30 [00:00<00:03,  8.67it/s]
2025-09-07T08:48:42.1668168Z running benchmark:   7% 2/30 [00:00<00:03,  8.77it/s]
2025-09-07T08:48:42.2803021Z running benchmark:  10% 3/30 [00:00<00:03,  8.85it/s]
2025-09-07T08:48:42.3912420Z running benchmark:  13% 4/30 [00:00<00:02,  8.84it/s]
2025-09-07T08:48:42.5023055Z running benchmark:  17% 5/30 [00:00<00:02,  8.90it/s]
2025-09-07T08:48:42.6156204Z running benchmark:  20% 6/30 [00:00<00:02,  8.94it/s]
2025-09-07T08:48:42.7279458Z running benchmark:  23% 7/30 [00:00<00:02,  8.90it/s]
2025-09-07T08:48:42.8399143Z running benchmark:  27% 8/30 [00:00<00:02,  8.90it/s]
2025-09-07T08:48:42.9529976Z running benchmark:  30% 9/30 [00:01<00:02,  8.91it/s]
2025-09-07T08:48:43.0659309Z running benchmark:  33% 10/30 [00:01<00:02,  8.89it/s]
2025-09-07T08:48:43.1771600Z running benchmark:  37% 11/30 [00:01<00:02,  8.88it/s]
2025-09-07T08:48:43.2898814Z running benchmark:  40% 12/30 [00:01<00:02,  8.91it/s]
2025-09-07T08:48:43.4024671Z running benchmark:  43% 13/30 [00:01<00:01,  8.90it/s]
2025-09-07T08:48:43.5153855Z running benchmark:  47% 14/30 [00:01<00:01,  8.89it/s]
2025-09-07T08:48:43.6279493Z running benchmark:  50% 15/30 [00:01<00:01,  8.88it/s]
2025-09-07T08:48:43.7407842Z running benchmark:  53% 16/30 [00:01<00:01,  8.88it/s]
2025-09-07T08:48:43.8535949Z running benchmark:  57% 17/30 [00:01<00:01,  8.88it/s]
2025-09-07T08:48:43.9648799Z running benchmark:  60% 18/30 [00:02<00:01,  8.87it/s]
2025-09-07T08:48:44.0756575Z running benchmark:  63% 19/30 [00:02<00:01,  8.91it/s]
2025-09-07T08:48:44.1861362Z running benchmark:  67% 20/30 [00:02<00:01,  8.94it/s]
2025-09-07T08:48:44.2976614Z running benchmark:  70% 21/30 [00:02<00:01,  8.97it/s]
2025-09-07T08:48:44.4085885Z running benchmark:  73% 22/30 [00:02<00:00,  8.97it/s]
2025-09-07T08:48:44.5197766Z running benchmark:  77% 23/30 [00:02<00:00,  8.98it/s]
2025-09-07T08:48:44.6332060Z running benchmark:  80% 24/30 [00:02<00:00,  8.99it/s]
2025-09-07T08:48:44.7461766Z running benchmark:  83% 25/30 [00:02<00:00,  8.94it/s]
2025-09-07T08:48:44.8577345Z running benchmark:  87% 26/30 [00:02<00:00,  8.91it/s]
2025-09-07T08:48:45.0389695Z running benchmark:  90% 27/30 [00:03<00:00,  8.93it/s]
2025-09-07T08:48:45.1509609Z running benchmark:  93% 28/30 [00:03<00:00,  7.53it/s]
2025-09-07T08:48:45.2626968Z running benchmark:  97% 29/30 [00:03<00:00,  7.90it/s]
2025-09-07T08:48:45.2627375Z running benchmark: 100% 30/30 [00:03<00:00,  8.19it/s]
2025-09-07T08:48:45.2627672Z running benchmark: 100% 30/30 [00:03<00:00,  8.73it/s]
2025-09-07T08:48:45.3450867Z 1.613x
2025-09-07T08:48:45.3454147Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:48:47.5185464Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:48:47.5186620Z   import pynvml  # type: ignore[import]
2025-09-07T08:48:49.5309953Z 
2025-09-07T08:48:49.9321873Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:48:49.9322145Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:48:49.9347628Z cpu  eval  functorch_dp_cifar10               
2025-09-07T08:49:11.7164639Z 
2025-09-07T08:49:11.8239025Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:49:11.9288941Z running benchmark:  20% 6/30 [00:00<00:00, 56.08it/s]
2025-09-07T08:49:12.0330599Z running benchmark:  40% 12/30 [00:00<00:00, 56.69it/s]
2025-09-07T08:49:12.1378921Z running benchmark:  60% 18/30 [00:00<00:00, 57.10it/s]
2025-09-07T08:49:12.2423587Z running benchmark:  80% 24/30 [00:00<00:00, 57.15it/s]
2025-09-07T08:49:12.2424057Z running benchmark: 100% 30/30 [00:00<00:00, 57.27it/s]
2025-09-07T08:49:12.2424366Z running benchmark: 100% 30/30 [00:00<00:00, 57.08it/s]
2025-09-07T08:49:12.2953089Z 0.980x
2025-09-07T08:49:12.2956081Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:49:14.3420304Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:49:14.3421185Z   import pynvml  # type: ignore[import]
2025-09-07T08:49:16.3522693Z 
2025-09-07T08:49:16.5361145Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:49:16.5361424Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:49:16.5365052Z cpu  eval  functorch_maml_omniglot            
2025-09-07T08:49:25.7994198Z 
2025-09-07T08:49:25.9012083Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:49:25.9012421Z running benchmark: 100% 30/30 [00:00<00:00, 296.05it/s]
2025-09-07T08:49:25.9013162Z running benchmark: 100% 30/30 [00:00<00:00, 295.54it/s]
2025-09-07T08:49:25.9481401Z 0.938x
2025-09-07T08:49:25.9484482Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:49:27.9017313Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:49:27.9018351Z   import pynvml  # type: ignore[import]
2025-09-07T08:49:29.9143277Z 
2025-09-07T08:49:31.4486123Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:49:31.4486401Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:49:31.4509051Z cpu  eval  hf_Albert                          
2025-09-07T08:49:45.4484200Z 
2025-09-07T08:49:45.6376390Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:49:45.8219003Z running benchmark:   7% 2/30 [00:00<00:02, 10.59it/s]
2025-09-07T08:49:46.0045248Z running benchmark:  13% 4/30 [00:00<00:02, 10.74it/s]
2025-09-07T08:49:46.2587544Z running benchmark:  20% 6/30 [00:00<00:02, 10.84it/s]
2025-09-07T08:49:46.4429709Z running benchmark:  27% 8/30 [00:00<00:02,  9.43it/s]
2025-09-07T08:49:46.6259464Z running benchmark:  33% 10/30 [00:00<00:02,  9.90it/s]
2025-09-07T08:49:46.8089044Z running benchmark:  40% 12/30 [00:01<00:01, 10.23it/s]
2025-09-07T08:49:46.9907758Z running benchmark:  47% 14/30 [00:01<00:01, 10.45it/s]
2025-09-07T08:49:47.1738622Z running benchmark:  53% 16/30 [00:01<00:01, 10.62it/s]
2025-09-07T08:49:47.3575409Z running benchmark:  60% 18/30 [00:01<00:01, 10.71it/s]
2025-09-07T08:49:47.5408283Z running benchmark:  67% 20/30 [00:01<00:00, 10.76it/s]
2025-09-07T08:49:47.7238160Z running benchmark:  73% 22/30 [00:02<00:00, 10.81it/s]
2025-09-07T08:49:47.9067838Z running benchmark:  80% 24/30 [00:02<00:00, 10.85it/s]
2025-09-07T08:49:48.0906581Z running benchmark:  87% 26/30 [00:02<00:00, 10.87it/s]
2025-09-07T08:49:48.2728764Z running benchmark:  93% 28/30 [00:02<00:00, 10.87it/s]
2025-09-07T08:49:48.2729152Z running benchmark: 100% 30/30 [00:02<00:00, 10.90it/s]
2025-09-07T08:49:48.2729887Z running benchmark: 100% 30/30 [00:02<00:00, 10.62it/s]
2025-09-07T08:49:48.3377758Z 1.347x
2025-09-07T08:49:48.3381539Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:49:50.5683535Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:49:50.5684732Z   import pynvml  # type: ignore[import]
2025-09-07T08:49:52.5789150Z 
2025-09-07T08:49:56.4965874Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:49:56.4966493Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:49:56.5207128Z cpu  eval  hf_Bart                            
2025-09-07T08:50:15.0224489Z 
2025-09-07T08:50:15.1612278Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:50:15.2963752Z running benchmark:   3% 1/30 [00:00<00:04,  7.22it/s]
2025-09-07T08:50:15.4309272Z running benchmark:   7% 2/30 [00:00<00:03,  7.33it/s]
2025-09-07T08:50:15.5637270Z running benchmark:  10% 3/30 [00:00<00:03,  7.37it/s]
2025-09-07T08:50:15.6994252Z running benchmark:  13% 4/30 [00:00<00:03,  7.43it/s]
2025-09-07T08:50:15.8329950Z running benchmark:  17% 5/30 [00:00<00:03,  7.41it/s]
2025-09-07T08:50:15.9686424Z running benchmark:  20% 6/30 [00:00<00:03,  7.44it/s]
2025-09-07T08:50:16.1042217Z running benchmark:  23% 7/30 [00:00<00:03,  7.42it/s]
2025-09-07T08:50:16.2371525Z running benchmark:  27% 8/30 [00:01<00:02,  7.40it/s]
2025-09-07T08:50:16.3706688Z running benchmark:  30% 9/30 [00:01<00:02,  7.44it/s]
2025-09-07T08:50:16.5050148Z running benchmark:  33% 10/30 [00:01<00:02,  7.45it/s]
2025-09-07T08:50:16.6391681Z running benchmark:  37% 11/30 [00:01<00:02,  7.45it/s]
2025-09-07T08:50:16.7740658Z running benchmark:  40% 12/30 [00:01<00:02,  7.45it/s]
2025-09-07T08:50:16.9077196Z running benchmark:  43% 13/30 [00:01<00:02,  7.44it/s]
2025-09-07T08:50:17.0421354Z running benchmark:  47% 14/30 [00:01<00:02,  7.45it/s]
2025-09-07T08:50:17.1759272Z running benchmark:  50% 15/30 [00:02<00:02,  7.45it/s]
2025-09-07T08:50:17.3101267Z running benchmark:  53% 16/30 [00:02<00:01,  7.46it/s]
2025-09-07T08:50:17.4461576Z running benchmark:  57% 17/30 [00:02<00:01,  7.46it/s]
2025-09-07T08:50:17.5814513Z running benchmark:  60% 18/30 [00:02<00:01,  7.42it/s]
2025-09-07T08:50:17.7159520Z running benchmark:  63% 19/30 [00:02<00:01,  7.41it/s]
2025-09-07T08:50:17.8503302Z running benchmark:  67% 20/30 [00:02<00:01,  7.42it/s]
2025-09-07T08:50:17.9843492Z running benchmark:  70% 21/30 [00:02<00:01,  7.43it/s]
2025-09-07T08:50:18.1193370Z running benchmark:  73% 22/30 [00:02<00:01,  7.44it/s]
2025-09-07T08:50:18.2537762Z running benchmark:  77% 23/30 [00:03<00:00,  7.43it/s]
2025-09-07T08:50:18.3891980Z running benchmark:  80% 24/30 [00:03<00:00,  7.43it/s]
2025-09-07T08:50:18.5264697Z running benchmark:  83% 25/30 [00:03<00:00,  7.42it/s]
2025-09-07T08:50:18.7319008Z running benchmark:  87% 26/30 [00:03<00:00,  7.38it/s]
2025-09-07T08:50:18.8671696Z running benchmark:  90% 27/30 [00:03<00:00,  6.39it/s]
2025-09-07T08:50:19.0035182Z running benchmark:  93% 28/30 [00:03<00:00,  6.66it/s]
2025-09-07T08:50:19.1383509Z running benchmark:  97% 29/30 [00:03<00:00,  6.85it/s]
2025-09-07T08:50:19.1383838Z running benchmark: 100% 30/30 [00:04<00:00,  7.01it/s]
2025-09-07T08:50:19.1384152Z running benchmark: 100% 30/30 [00:04<00:00,  7.29it/s]
2025-09-07T08:50:19.2379944Z 1.206x
2025-09-07T08:50:19.2384358Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:50:21.6538166Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:50:21.6539028Z   import pynvml  # type: ignore[import]
2025-09-07T08:50:23.6587263Z 
2025-09-07T08:50:26.5262832Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:50:26.5263358Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:50:26.5448179Z cpu  eval  hf_Bert                            
2025-09-07T08:50:39.8704727Z 
2025-09-07T08:50:39.9767984Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:50:40.0806674Z running benchmark:   3% 1/30 [00:00<00:03,  9.43it/s]
2025-09-07T08:50:40.1836433Z running benchmark:   7% 2/30 [00:00<00:02,  9.55it/s]
2025-09-07T08:50:40.2869480Z running benchmark:  10% 3/30 [00:00<00:02,  9.62it/s]
2025-09-07T08:50:40.3893745Z running benchmark:  13% 4/30 [00:00<00:02,  9.64it/s]
2025-09-07T08:50:40.4917872Z running benchmark:  17% 5/30 [00:00<00:02,  9.68it/s]
2025-09-07T08:50:40.5941593Z running benchmark:  20% 6/30 [00:00<00:02,  9.71it/s]
2025-09-07T08:50:40.6978421Z running benchmark:  23% 7/30 [00:00<00:02,  9.73it/s]
2025-09-07T08:50:40.8009389Z running benchmark:  27% 8/30 [00:00<00:02,  9.70it/s]
2025-09-07T08:50:40.9052584Z running benchmark:  30% 9/30 [00:00<00:02,  9.70it/s]
2025-09-07T08:50:41.0092177Z running benchmark:  33% 10/30 [00:01<00:02,  9.66it/s]
2025-09-07T08:50:41.1128637Z running benchmark:  37% 11/30 [00:01<00:01,  9.65it/s]
2025-09-07T08:50:41.2167144Z running benchmark:  40% 12/30 [00:01<00:01,  9.65it/s]
2025-09-07T08:50:41.3199025Z running benchmark:  43% 13/30 [00:01<00:01,  9.64it/s]
2025-09-07T08:50:41.4235158Z running benchmark:  47% 14/30 [00:01<00:01,  9.66it/s]
2025-09-07T08:50:41.5264205Z running benchmark:  50% 15/30 [00:01<00:01,  9.66it/s]
2025-09-07T08:50:41.6278964Z running benchmark:  53% 16/30 [00:01<00:01,  9.67it/s]
2025-09-07T08:50:41.7307149Z running benchmark:  57% 17/30 [00:01<00:01,  9.73it/s]
2025-09-07T08:50:41.8343360Z running benchmark:  60% 18/30 [00:01<00:01,  9.73it/s]
2025-09-07T08:50:41.9378115Z running benchmark:  63% 19/30 [00:01<00:01,  9.70it/s]
2025-09-07T08:50:42.0399496Z running benchmark:  67% 20/30 [00:02<00:01,  9.70it/s]
2025-09-07T08:50:42.1430481Z running benchmark:  70% 21/30 [00:02<00:00,  9.72it/s]
2025-09-07T08:50:42.2467986Z running benchmark:  73% 22/30 [00:02<00:00,  9.71it/s]
2025-09-07T08:50:42.3505340Z running benchmark:  77% 23/30 [00:02<00:00,  9.69it/s]
2025-09-07T08:50:42.4532355Z running benchmark:  80% 24/30 [00:02<00:00,  9.68it/s]
2025-09-07T08:50:42.5550639Z running benchmark:  83% 25/30 [00:02<00:00,  9.69it/s]
2025-09-07T08:50:42.6569289Z running benchmark:  87% 26/30 [00:02<00:00,  9.73it/s]
2025-09-07T08:50:42.7603319Z running benchmark:  90% 27/30 [00:02<00:00,  9.76it/s]
2025-09-07T08:50:42.8655949Z running benchmark:  93% 28/30 [00:02<00:00,  9.73it/s]
2025-09-07T08:50:42.9695919Z running benchmark:  97% 29/30 [00:02<00:00,  9.66it/s]
2025-09-07T08:50:42.9697165Z running benchmark: 100% 30/30 [00:03<00:00,  9.65it/s]
2025-09-07T08:50:42.9698608Z running benchmark: 100% 30/30 [00:03<00:00,  9.68it/s]
2025-09-07T08:50:43.0612703Z 1.148x
2025-09-07T08:50:43.0616703Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:50:45.4310719Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:50:45.4311830Z   import pynvml  # type: ignore[import]
2025-09-07T08:50:47.4510564Z 
2025-09-07T08:50:52.0447362Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:50:52.0447703Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:50:52.0775846Z cpu  eval  hf_Bert_large                      
2025-09-07T08:51:11.7631078Z 
2025-09-07T08:51:11.9924240Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:51:12.2315332Z running benchmark:   3% 1/30 [00:00<00:06,  4.36it/s]
2025-09-07T08:51:12.4690041Z running benchmark:   7% 2/30 [00:00<00:06,  4.26it/s]
2025-09-07T08:51:12.7068554Z running benchmark:  10% 3/30 [00:00<00:06,  4.23it/s]
2025-09-07T08:51:12.9446499Z running benchmark:  13% 4/30 [00:00<00:06,  4.22it/s]
2025-09-07T08:51:13.1821138Z running benchmark:  17% 5/30 [00:01<00:05,  4.22it/s]
2025-09-07T08:51:13.4178513Z running benchmark:  20% 6/30 [00:01<00:05,  4.21it/s]
2025-09-07T08:51:13.6545502Z running benchmark:  23% 7/30 [00:01<00:05,  4.22it/s]
2025-09-07T08:51:13.8934742Z running benchmark:  27% 8/30 [00:01<00:05,  4.22it/s]
2025-09-07T08:51:14.1305598Z running benchmark:  30% 9/30 [00:02<00:04,  4.21it/s]
2025-09-07T08:51:14.3663505Z running benchmark:  33% 10/30 [00:02<00:04,  4.21it/s]
2025-09-07T08:51:14.6032491Z running benchmark:  37% 11/30 [00:02<00:04,  4.22it/s]
2025-09-07T08:51:14.8392366Z running benchmark:  40% 12/30 [00:02<00:04,  4.22it/s]
2025-09-07T08:51:15.0766671Z running benchmark:  43% 13/30 [00:03<00:04,  4.23it/s]
2025-09-07T08:51:15.3115044Z running benchmark:  47% 14/30 [00:03<00:03,  4.22it/s]
2025-09-07T08:51:15.5479245Z running benchmark:  50% 15/30 [00:03<00:03,  4.23it/s]
2025-09-07T08:51:15.8537253Z running benchmark:  53% 16/30 [00:03<00:03,  4.23it/s]
2025-09-07T08:51:16.0906966Z running benchmark:  57% 17/30 [00:04<00:03,  3.89it/s]
2025-09-07T08:51:16.3272314Z running benchmark:  60% 18/30 [00:04<00:03,  3.98it/s]
2025-09-07T08:51:16.5661010Z running benchmark:  63% 19/30 [00:04<00:02,  4.05it/s]
2025-09-07T08:51:16.8024916Z running benchmark:  67% 20/30 [00:04<00:02,  4.09it/s]
2025-09-07T08:51:17.0397501Z running benchmark:  70% 21/30 [00:05<00:02,  4.13it/s]
2025-09-07T08:51:17.2712937Z running benchmark:  73% 22/30 [00:05<00:01,  4.16it/s]
2025-09-07T08:51:17.5080245Z running benchmark:  77% 23/30 [00:05<00:01,  4.20it/s]
2025-09-07T08:51:17.7437367Z running benchmark:  80% 24/30 [00:05<00:01,  4.21it/s]
2025-09-07T08:51:17.9800893Z running benchmark:  83% 25/30 [00:05<00:01,  4.22it/s]
2025-09-07T08:51:18.2157610Z running benchmark:  87% 26/30 [00:06<00:00,  4.22it/s]
2025-09-07T08:51:18.4549483Z running benchmark:  90% 27/30 [00:06<00:00,  4.23it/s]
2025-09-07T08:51:18.6953343Z running benchmark:  93% 28/30 [00:06<00:00,  4.21it/s]
2025-09-07T08:51:18.9318879Z running benchmark:  97% 29/30 [00:06<00:00,  4.20it/s]
2025-09-07T08:51:18.9319351Z running benchmark: 100% 30/30 [00:07<00:00,  4.21it/s]
2025-09-07T08:51:18.9319690Z running benchmark: 100% 30/30 [00:07<00:00,  4.18it/s]
2025-09-07T08:51:19.0658405Z 1.169x
2025-09-07T08:51:19.0663014Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:51:21.7227636Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:51:21.7228473Z   import pynvml  # type: ignore[import]
2025-09-07T08:51:23.7383280Z 
2025-09-07T08:51:25.1448498Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:51:25.1449863Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:51:25.1450691Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:51:25.1451360Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:51:25.1452634Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T08:51:25.1454125Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T08:51:25.1455023Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T08:51:25.1461913Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T08:51:26.8179063Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:51:26.8179752Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T08:51:27.3584139Z 
2025-09-07T08:51:27.3585875Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:51:27.3793773Z cpu  eval  hf_BigBird                         
2025-09-07T08:51:59.9881153Z W0907 08:51:59.987297 140020 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] torch._dynamo hit config.recompile_limit (8)
2025-09-07T08:51:59.9882468Z W0907 08:51:59.987297 140020 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    function: 'forward' (/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py:417)
2025-09-07T08:51:59.9884329Z W0907 08:51:59.987297 140020 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    last reason: 10/7: self.seed == 7                                           # context_layer, attention_probs = self.bigbird_block_sparse_attention(  # transformers/models/big_bird/modeling_big_bird.py:455 in forward (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
2025-09-07T08:51:59.9885929Z W0907 08:51:59.987297 140020 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-09-07T08:51:59.9886770Z W0907 08:51:59.987297 140020 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2025-09-07T08:52:03.4815427Z 
2025-09-07T08:52:04.3701769Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:52:05.3257530Z running benchmark:   3% 1/30 [00:00<00:25,  1.13it/s]
2025-09-07T08:52:06.2060282Z running benchmark:   7% 2/30 [00:01<00:25,  1.08it/s]
2025-09-07T08:52:07.0861010Z running benchmark:  10% 3/30 [00:02<00:24,  1.10it/s]
2025-09-07T08:52:07.9653918Z running benchmark:  13% 4/30 [00:03<00:23,  1.12it/s]
2025-09-07T08:52:08.9203688Z running benchmark:  17% 5/30 [00:04<00:22,  1.12it/s]
2025-09-07T08:52:09.7975633Z running benchmark:  20% 6/30 [00:05<00:21,  1.10it/s]
2025-09-07T08:52:10.6797272Z running benchmark:  23% 7/30 [00:06<00:20,  1.11it/s]
2025-09-07T08:52:11.6299873Z running benchmark:  27% 8/30 [00:07<00:19,  1.12it/s]
2025-09-07T08:52:12.5072757Z running benchmark:  30% 9/30 [00:08<00:19,  1.10it/s]
2025-09-07T08:52:13.4578021Z running benchmark:  33% 10/30 [00:09<00:18,  1.11it/s]
2025-09-07T08:52:14.3375357Z running benchmark:  37% 11/30 [00:09<00:17,  1.09it/s]
2025-09-07T08:52:15.2185324Z running benchmark:  40% 12/30 [00:10<00:16,  1.10it/s]
2025-09-07T08:52:16.0949191Z running benchmark:  43% 13/30 [00:11<00:15,  1.11it/s]
2025-09-07T08:52:17.0451611Z running benchmark:  47% 14/30 [00:12<00:14,  1.12it/s]
2025-09-07T08:52:17.9263607Z running benchmark:  50% 15/30 [00:13<00:13,  1.10it/s]
2025-09-07T08:52:18.8088279Z running benchmark:  53% 16/30 [00:14<00:12,  1.11it/s]
2025-09-07T08:52:19.6950219Z running benchmark:  57% 17/30 [00:15<00:11,  1.12it/s]
2025-09-07T08:52:20.5810430Z running benchmark:  60% 18/30 [00:16<00:10,  1.12it/s]
2025-09-07T08:52:21.5378593Z running benchmark:  63% 19/30 [00:17<00:09,  1.12it/s]
2025-09-07T08:52:22.4230746Z running benchmark:  67% 20/30 [00:18<00:09,  1.10it/s]
2025-09-07T08:52:23.3035683Z running benchmark:  70% 21/30 [00:18<00:08,  1.11it/s]
2025-09-07T08:52:24.1781354Z running benchmark:  73% 22/30 [00:19<00:07,  1.12it/s]
2025-09-07T08:52:25.1310915Z running benchmark:  77% 23/30 [00:20<00:06,  1.12it/s]
2025-09-07T08:52:26.0148661Z running benchmark:  80% 24/30 [00:21<00:05,  1.10it/s]
2025-09-07T08:52:26.9011732Z running benchmark:  83% 25/30 [00:22<00:04,  1.11it/s]
2025-09-07T08:52:27.7852207Z running benchmark:  87% 26/30 [00:23<00:03,  1.12it/s]
2025-09-07T08:52:28.6727376Z running benchmark:  90% 27/30 [00:24<00:02,  1.12it/s]
2025-09-07T08:52:29.6230398Z running benchmark:  93% 28/30 [00:25<00:01,  1.12it/s]
2025-09-07T08:52:30.5101177Z running benchmark:  97% 29/30 [00:26<00:00,  1.10it/s]
2025-09-07T08:52:30.5101499Z running benchmark: 100% 30/30 [00:27<00:00,  1.11it/s]
2025-09-07T08:52:30.5101762Z running benchmark: 100% 30/30 [00:27<00:00,  1.11it/s]
2025-09-07T08:52:30.6156549Z 1.059x
2025-09-07T08:52:30.6161297Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:52:33.1247220Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:52:33.1248107Z   import pynvml  # type: ignore[import]
2025-09-07T08:52:35.1482473Z 
2025-09-07T08:52:37.6243522Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:52:37.6243858Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:52:37.6336464Z cpu  eval  hf_DistilBert                      
2025-09-07T08:52:47.7009678Z 
2025-09-07T08:52:47.8189531Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:52:47.9324843Z running benchmark:   7% 2/30 [00:00<00:01, 16.98it/s]
2025-09-07T08:52:48.0442993Z running benchmark:  13% 4/30 [00:00<00:01, 17.35it/s]
2025-09-07T08:52:48.1582585Z running benchmark:  20% 6/30 [00:00<00:01, 17.59it/s]
2025-09-07T08:52:48.2704353Z running benchmark:  27% 8/30 [00:00<00:01, 17.57it/s]
2025-09-07T08:52:48.3840117Z running benchmark:  33% 10/30 [00:00<00:01, 17.67it/s]
2025-09-07T08:52:48.5001570Z running benchmark:  40% 12/30 [00:00<00:01, 17.64it/s]
2025-09-07T08:52:48.6127771Z running benchmark:  47% 14/30 [00:00<00:00, 17.50it/s]
2025-09-07T08:52:48.7256686Z running benchmark:  53% 16/30 [00:00<00:00, 17.59it/s]
2025-09-07T08:52:48.8389522Z running benchmark:  60% 18/30 [00:01<00:00, 17.62it/s]
2025-09-07T08:52:48.9512346Z running benchmark:  67% 20/30 [00:01<00:00, 17.64it/s]
2025-09-07T08:52:49.0633595Z running benchmark:  73% 22/30 [00:01<00:00, 17.69it/s]
2025-09-07T08:52:49.1757616Z running benchmark:  80% 24/30 [00:01<00:00, 17.73it/s]
2025-09-07T08:52:49.2859657Z running benchmark:  87% 26/30 [00:01<00:00, 17.75it/s]
2025-09-07T08:52:49.4707682Z running benchmark:  93% 28/30 [00:01<00:00, 17.87it/s]
2025-09-07T08:52:49.4708028Z running benchmark: 100% 30/30 [00:01<00:00, 14.94it/s]
2025-09-07T08:52:49.4708349Z running benchmark: 100% 30/30 [00:01<00:00, 16.95it/s]
2025-09-07T08:52:49.5451454Z 1.168x
2025-09-07T08:52:49.5455214Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:52:51.8380530Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:52:51.8381449Z   import pynvml  # type: ignore[import]
2025-09-07T08:52:53.8555726Z 
2025-09-07T08:52:58.3676814Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:52:58.3677178Z loading model: 0it [00:04, ?it/s]
2025-09-07T08:52:58.3824340Z cpu  eval  hf_GPT2                            
2025-09-07T08:53:18.0382032Z 
2025-09-07T08:53:18.2045185Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:53:18.3647719Z running benchmark:   7% 2/30 [00:00<00:02, 12.04it/s]
2025-09-07T08:53:18.5305566Z running benchmark:  13% 4/30 [00:00<00:02, 12.29it/s]
2025-09-07T08:53:18.6923743Z running benchmark:  20% 6/30 [00:00<00:01, 12.19it/s]
2025-09-07T08:53:18.8532786Z running benchmark:  27% 8/30 [00:00<00:01, 12.26it/s]
2025-09-07T08:53:19.0153446Z running benchmark:  33% 10/30 [00:00<00:01, 12.32it/s]
2025-09-07T08:53:19.1769007Z running benchmark:  40% 12/30 [00:00<00:01, 12.33it/s]
2025-09-07T08:53:19.3373143Z running benchmark:  47% 14/30 [00:01<00:01, 12.34it/s]
2025-09-07T08:53:19.4995546Z running benchmark:  53% 16/30 [00:01<00:01, 12.38it/s]
2025-09-07T08:53:19.6602778Z running benchmark:  60% 18/30 [00:01<00:00, 12.37it/s]
2025-09-07T08:53:19.8216816Z running benchmark:  67% 20/30 [00:01<00:00, 12.39it/s]
2025-09-07T08:53:19.9843983Z running benchmark:  73% 22/30 [00:01<00:00, 12.39it/s]
2025-09-07T08:53:20.1455361Z running benchmark:  80% 24/30 [00:01<00:00, 12.36it/s]
2025-09-07T08:53:20.3062196Z running benchmark:  87% 26/30 [00:02<00:00, 12.38it/s]
2025-09-07T08:53:20.4680571Z running benchmark:  93% 28/30 [00:02<00:00, 12.40it/s]
2025-09-07T08:53:20.4680927Z running benchmark: 100% 30/30 [00:02<00:00, 12.39it/s]
2025-09-07T08:53:20.4681220Z running benchmark: 100% 30/30 [00:02<00:00, 12.35it/s]
2025-09-07T08:53:20.5628511Z 1.306x
2025-09-07T08:53:20.5633209Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:53:23.0752910Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:53:23.0753787Z   import pynvml  # type: ignore[import]
2025-09-07T08:53:25.0751535Z 
2025-09-07T08:53:37.1014745Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:53:37.1015478Z loading model: 0it [00:12, ?it/s]
2025-09-07T08:53:37.1610759Z cpu  eval  hf_GPT2_large                      
2025-09-07T08:54:12.9181113Z 
2025-09-07T08:54:13.9071457Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:54:14.8859958Z running benchmark:   3% 1/30 [00:00<00:28,  1.01it/s]
2025-09-07T08:54:15.9310010Z running benchmark:   7% 2/30 [00:01<00:27,  1.02it/s]
2025-09-07T08:54:16.9112762Z running benchmark:  10% 3/30 [00:03<00:27,  1.01s/it]
2025-09-07T08:54:17.8985372Z running benchmark:  13% 4/30 [00:03<00:25,  1.00it/s]
2025-09-07T08:54:18.8819639Z running benchmark:  17% 5/30 [00:04<00:24,  1.01it/s]
2025-09-07T08:54:19.9355089Z running benchmark:  20% 6/30 [00:05<00:23,  1.01it/s]
2025-09-07T08:54:20.9226832Z running benchmark:  23% 7/30 [00:07<00:23,  1.01s/it]
2025-09-07T08:54:21.9068090Z running benchmark:  27% 8/30 [00:08<00:22,  1.00s/it]
2025-09-07T08:54:22.8934155Z running benchmark:  30% 9/30 [00:08<00:20,  1.00it/s]
2025-09-07T08:54:23.9426053Z running benchmark:  33% 10/30 [00:09<00:19,  1.01it/s]
2025-09-07T08:54:24.9291876Z running benchmark:  37% 11/30 [00:11<00:19,  1.01s/it]
2025-09-07T08:54:25.9154469Z running benchmark:  40% 12/30 [00:12<00:18,  1.00s/it]
2025-09-07T08:54:26.8949412Z running benchmark:  43% 13/30 [00:12<00:16,  1.00it/s]
2025-09-07T08:54:27.9585448Z running benchmark:  47% 14/30 [00:13<00:15,  1.01it/s]
2025-09-07T08:54:28.9473205Z running benchmark:  50% 15/30 [00:15<00:15,  1.01s/it]
2025-09-07T08:54:29.9302271Z running benchmark:  53% 16/30 [00:16<00:14,  1.01s/it]
2025-09-07T08:54:30.9124551Z running benchmark:  57% 17/30 [00:17<00:12,  1.00it/s]
2025-09-07T08:54:31.8941299Z running benchmark:  60% 18/30 [00:17<00:11,  1.01it/s]
2025-09-07T08:54:32.8781483Z running benchmark:  63% 19/30 [00:18<00:10,  1.01it/s]
2025-09-07T08:54:33.8557067Z running benchmark:  67% 20/30 [00:19<00:09,  1.01it/s]
2025-09-07T08:54:34.8352367Z running benchmark:  70% 21/30 [00:20<00:08,  1.01it/s]
2025-09-07T08:54:35.8194620Z running benchmark:  73% 22/30 [00:21<00:07,  1.02it/s]
2025-09-07T08:54:36.8039646Z running benchmark:  77% 23/30 [00:22<00:06,  1.02it/s]
2025-09-07T08:54:37.7887797Z running benchmark:  80% 24/30 [00:23<00:05,  1.02it/s]
2025-09-07T08:54:38.7773679Z running benchmark:  83% 25/30 [00:24<00:04,  1.02it/s]
2025-09-07T08:54:39.7570415Z running benchmark:  87% 26/30 [00:25<00:03,  1.01it/s]
2025-09-07T08:54:40.7395486Z running benchmark:  90% 27/30 [00:26<00:02,  1.02it/s]
2025-09-07T08:54:41.7276530Z running benchmark:  93% 28/30 [00:27<00:01,  1.02it/s]
2025-09-07T08:54:42.7147216Z running benchmark:  97% 29/30 [00:28<00:00,  1.02it/s]
2025-09-07T08:54:42.7147520Z running benchmark: 100% 30/30 [00:29<00:00,  1.01it/s]
2025-09-07T08:54:42.7147756Z running benchmark: 100% 30/30 [00:29<00:00,  1.01it/s]
2025-09-07T08:54:42.9491820Z 0.982x
2025-09-07T08:54:42.9496349Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:54:46.1185449Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:54:46.1186312Z   import pynvml  # type: ignore[import]
2025-09-07T08:54:48.1387967Z 
2025-09-07T08:54:51.6484163Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:54:51.6487094Z loading model: 0it [00:03, ?it/s]
2025-09-07T08:54:51.6717607Z cpu  eval  hf_Longformer                      
2025-09-07T08:54:52.8674581Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:54:52.8675263Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:54:52.8675792Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T08:54:52.8676292Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:54:52.8676890Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T08:54:52.8677370Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:54:52.8677824Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T08:54:52.8678502Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T08:54:52.8679172Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(**inputs)
2025-09-07T08:54:52.8679949Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T08:54:52.8680763Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.longformer(
2025-09-07T08:54:52.8681525Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T08:54:52.8682280Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     encoder_outputs = self.encoder(
2025-09-07T08:54:52.8683029Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1244, in forward
2025-09-07T08:54:52.8683928Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2025-09-07T08:54:52.8684517Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:54:52.8684895Z W0907 08:54:52.866622 145613 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T08:55:55.1487684Z 
2025-09-07T08:55:55.3896857Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:55:55.6285501Z running benchmark:   3% 1/30 [00:00<00:06,  4.15it/s]
2025-09-07T08:55:55.8650274Z running benchmark:   7% 2/30 [00:00<00:06,  4.17it/s]
2025-09-07T08:55:56.1046253Z running benchmark:  10% 3/30 [00:00<00:06,  4.20it/s]
2025-09-07T08:55:56.3406504Z running benchmark:  13% 4/30 [00:00<00:06,  4.19it/s]
2025-09-07T08:55:56.5776278Z running benchmark:  17% 5/30 [00:01<00:05,  4.21it/s]
2025-09-07T08:55:56.8144314Z running benchmark:  20% 6/30 [00:01<00:05,  4.21it/s]
2025-09-07T08:55:57.0515820Z running benchmark:  23% 7/30 [00:01<00:05,  4.21it/s]
2025-09-07T08:55:57.2868288Z running benchmark:  27% 8/30 [00:01<00:05,  4.22it/s]
2025-09-07T08:55:57.5213708Z running benchmark:  30% 9/30 [00:02<00:04,  4.23it/s]
2025-09-07T08:55:57.7597792Z running benchmark:  33% 10/30 [00:02<00:04,  4.24it/s]
2025-09-07T08:55:57.9973771Z running benchmark:  37% 11/30 [00:02<00:04,  4.22it/s]
2025-09-07T08:55:58.2326501Z running benchmark:  40% 12/30 [00:02<00:04,  4.22it/s]
2025-09-07T08:55:58.4735143Z running benchmark:  43% 13/30 [00:03<00:04,  4.23it/s]
2025-09-07T08:55:58.7115682Z running benchmark:  47% 14/30 [00:03<00:03,  4.21it/s]
2025-09-07T08:55:58.9485982Z running benchmark:  50% 15/30 [00:03<00:03,  4.20it/s]
2025-09-07T08:55:59.1844769Z running benchmark:  53% 16/30 [00:03<00:03,  4.21it/s]
2025-09-07T08:55:59.4205360Z running benchmark:  57% 17/30 [00:04<00:03,  4.22it/s]
2025-09-07T08:55:59.6574260Z running benchmark:  60% 18/30 [00:04<00:02,  4.22it/s]
2025-09-07T08:55:59.8944939Z running benchmark:  63% 19/30 [00:04<00:02,  4.22it/s]
2025-09-07T08:56:00.1310297Z running benchmark:  67% 20/30 [00:04<00:02,  4.22it/s]
2025-09-07T08:56:00.3688224Z running benchmark:  70% 21/30 [00:04<00:02,  4.22it/s]
2025-09-07T08:56:00.6046561Z running benchmark:  73% 22/30 [00:05<00:01,  4.22it/s]
2025-09-07T08:56:00.8424761Z running benchmark:  77% 23/30 [00:05<00:01,  4.22it/s]
2025-09-07T08:56:01.0799620Z running benchmark:  80% 24/30 [00:05<00:01,  4.22it/s]
2025-09-07T08:56:01.3173959Z running benchmark:  83% 25/30 [00:05<00:01,  4.22it/s]
2025-09-07T08:56:01.5560173Z running benchmark:  87% 26/30 [00:06<00:00,  4.22it/s]
2025-09-07T08:56:01.7948108Z running benchmark:  90% 27/30 [00:06<00:00,  4.21it/s]
2025-09-07T08:56:02.0315509Z running benchmark:  93% 28/30 [00:06<00:00,  4.20it/s]
2025-09-07T08:56:02.2687885Z running benchmark:  97% 29/30 [00:06<00:00,  4.21it/s]
2025-09-07T08:56:02.2688209Z running benchmark: 100% 30/30 [00:07<00:00,  4.21it/s]
2025-09-07T08:56:02.2688481Z running benchmark: 100% 30/30 [00:07<00:00,  4.21it/s]
2025-09-07T08:56:02.3744088Z 1.417x
2025-09-07T08:56:02.3744877Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:56:05.2870516Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:56:05.2871348Z   import pynvml  # type: ignore[import]
2025-09-07T08:56:07.3068828Z 
2025-09-07T08:56:08.6534975Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:56:08.6536733Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T08:56:08.8881912Z 
2025-09-07T08:56:08.8882204Z loading model: 0it [00:01, ?it/s]
2025-09-07T08:56:08.8922888Z cpu  eval  hf_Reformer                        
2025-09-07T08:56:16.8049522Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T08:56:16.8050222Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T08:56:16.8050748Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] or:
2025-09-07T08:56:16.8051262Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T08:56:16.8051854Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] to include these operations in the captured graph.
2025-09-07T08:56:16.8052675Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:56:16.8053141Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break: from user code at:
2025-09-07T08:56:16.8054076Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2132, in torch_dynamo_resume_in_forward_at_2104
2025-09-07T08:56:16.8054969Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     embedding_output = self.embeddings(
2025-09-07T08:56:16.8055715Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 369, in forward
2025-09-07T08:56:16.8056517Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     position_embeddings = self.position_embeddings(position_ids)
2025-09-07T08:56:16.8057312Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T08:56:16.8058129Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     max_position_id = position_ids.max().item()
2025-09-07T08:56:16.8058602Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:56:16.8058977Z W0907 08:56:16.804013 146764 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T08:56:37.7141089Z 
2025-09-07T08:56:38.1326464Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:56:38.5492331Z running benchmark:   3% 1/30 [00:00<00:12,  2.39it/s]
2025-09-07T08:56:38.9624873Z running benchmark:   7% 2/30 [00:00<00:11,  2.40it/s]
2025-09-07T08:56:39.3747285Z running benchmark:  10% 3/30 [00:01<00:11,  2.41it/s]
2025-09-07T08:56:39.7870518Z running benchmark:  13% 4/30 [00:01<00:10,  2.41it/s]
2025-09-07T08:56:40.1997832Z running benchmark:  17% 5/30 [00:02<00:10,  2.42it/s]
2025-09-07T08:56:40.6122307Z running benchmark:  20% 6/30 [00:02<00:09,  2.42it/s]
2025-09-07T08:56:41.0252279Z running benchmark:  23% 7/30 [00:02<00:09,  2.42it/s]
2025-09-07T08:56:41.4379391Z running benchmark:  27% 8/30 [00:03<00:09,  2.42it/s]
2025-09-07T08:56:41.8539232Z running benchmark:  30% 9/30 [00:03<00:08,  2.42it/s]
2025-09-07T08:56:42.2687449Z running benchmark:  33% 10/30 [00:04<00:08,  2.42it/s]
2025-09-07T08:56:42.6820191Z running benchmark:  37% 11/30 [00:04<00:07,  2.41it/s]
2025-09-07T08:56:43.0962810Z running benchmark:  40% 12/30 [00:04<00:07,  2.42it/s]
2025-09-07T08:56:43.5099088Z running benchmark:  43% 13/30 [00:05<00:07,  2.42it/s]
2025-09-07T08:56:43.9243263Z running benchmark:  47% 14/30 [00:05<00:06,  2.42it/s]
2025-09-07T08:56:44.3389530Z running benchmark:  50% 15/30 [00:06<00:06,  2.42it/s]
2025-09-07T08:56:44.7523817Z running benchmark:  53% 16/30 [00:06<00:05,  2.41it/s]
2025-09-07T08:56:45.1653588Z running benchmark:  57% 17/30 [00:07<00:05,  2.42it/s]
2025-09-07T08:56:45.5802195Z running benchmark:  60% 18/30 [00:07<00:04,  2.42it/s]
2025-09-07T08:56:45.9961427Z running benchmark:  63% 19/30 [00:07<00:04,  2.42it/s]
2025-09-07T08:56:46.4095754Z running benchmark:  67% 20/30 [00:08<00:04,  2.41it/s]
2025-09-07T08:56:46.8229138Z running benchmark:  70% 21/30 [00:08<00:03,  2.41it/s]
2025-09-07T08:56:47.2357682Z running benchmark:  73% 22/30 [00:09<00:03,  2.42it/s]
2025-09-07T08:56:47.6547917Z running benchmark:  77% 23/30 [00:09<00:02,  2.42it/s]
2025-09-07T08:56:48.0688856Z running benchmark:  80% 24/30 [00:09<00:02,  2.41it/s]
2025-09-07T08:56:48.4861601Z running benchmark:  83% 25/30 [00:10<00:02,  2.41it/s]
2025-09-07T08:56:48.9000396Z running benchmark:  87% 26/30 [00:10<00:01,  2.41it/s]
2025-09-07T08:56:49.3137799Z running benchmark:  90% 27/30 [00:11<00:01,  2.41it/s]
2025-09-07T08:56:49.7272834Z running benchmark:  93% 28/30 [00:11<00:00,  2.41it/s]
2025-09-07T08:56:50.1395720Z running benchmark:  97% 29/30 [00:12<00:00,  2.41it/s]
2025-09-07T08:56:50.1399227Z running benchmark: 100% 30/30 [00:12<00:00,  2.42it/s]
2025-09-07T08:56:50.1400412Z running benchmark: 100% 30/30 [00:12<00:00,  2.41it/s]
2025-09-07T08:56:50.2024571Z 1.146x
2025-09-07T08:56:50.2025422Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:56:52.4459497Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:56:52.4460381Z   import pynvml  # type: ignore[import]
2025-09-07T08:56:54.4653528Z 
2025-09-07T08:57:00.2822913Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:57:00.2824533Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:57:00.3036042Z cpu  eval  hf_Roberta_base                    
2025-09-07T08:57:13.6263574Z 
2025-09-07T08:57:13.8127647Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:57:13.9952632Z running benchmark:   3% 1/30 [00:00<00:05,  5.37it/s]
2025-09-07T08:57:14.1675995Z running benchmark:   7% 2/30 [00:00<00:05,  5.43it/s]
2025-09-07T08:57:14.3364934Z running benchmark:  10% 3/30 [00:00<00:04,  5.60it/s]
2025-09-07T08:57:14.5016089Z running benchmark:  13% 4/30 [00:00<00:04,  5.72it/s]
2025-09-07T08:57:14.6748617Z running benchmark:  17% 5/30 [00:00<00:04,  5.84it/s]
2025-09-07T08:57:14.8458765Z running benchmark:  20% 6/30 [00:01<00:04,  5.81it/s]
2025-09-07T08:57:15.0162591Z running benchmark:  23% 7/30 [00:01<00:03,  5.83it/s]
2025-09-07T08:57:15.1867547Z running benchmark:  27% 8/30 [00:01<00:03,  5.84it/s]
2025-09-07T08:57:15.3597268Z running benchmark:  30% 9/30 [00:01<00:03,  5.85it/s]
2025-09-07T08:57:15.5353040Z running benchmark:  33% 10/30 [00:01<00:03,  5.83it/s]
2025-09-07T08:57:15.7058964Z running benchmark:  37% 11/30 [00:01<00:03,  5.79it/s]
2025-09-07T08:57:15.8776834Z running benchmark:  40% 12/30 [00:02<00:03,  5.81it/s]
2025-09-07T08:57:16.0468438Z running benchmark:  43% 13/30 [00:02<00:02,  5.81it/s]
2025-09-07T08:57:16.2165297Z running benchmark:  47% 14/30 [00:02<00:02,  5.84it/s]
2025-09-07T08:57:16.3862150Z running benchmark:  50% 15/30 [00:02<00:02,  5.86it/s]
2025-09-07T08:57:16.5551942Z running benchmark:  53% 16/30 [00:02<00:02,  5.87it/s]
2025-09-07T08:57:16.7275206Z running benchmark:  57% 17/30 [00:02<00:02,  5.88it/s]
2025-09-07T08:57:16.8959869Z running benchmark:  60% 18/30 [00:03<00:02,  5.86it/s]
2025-09-07T08:57:17.0652913Z running benchmark:  63% 19/30 [00:03<00:01,  5.88it/s]
2025-09-07T08:57:17.2344345Z running benchmark:  67% 20/30 [00:03<00:01,  5.89it/s]
2025-09-07T08:57:17.4048175Z running benchmark:  70% 21/30 [00:03<00:01,  5.90it/s]
2025-09-07T08:57:17.5761094Z running benchmark:  73% 22/30 [00:03<00:01,  5.89it/s]
2025-09-07T08:57:17.7464763Z running benchmark:  77% 23/30 [00:03<00:01,  5.87it/s]
2025-09-07T08:57:17.9137084Z running benchmark:  80% 24/30 [00:04<00:01,  5.87it/s]
2025-09-07T08:57:18.0837627Z running benchmark:  83% 25/30 [00:04<00:00,  5.90it/s]
2025-09-07T08:57:18.2558406Z running benchmark:  87% 26/30 [00:04<00:00,  5.90it/s]
2025-09-07T08:57:18.4266511Z running benchmark:  90% 27/30 [00:04<00:00,  5.87it/s]
2025-09-07T08:57:18.5982896Z running benchmark:  93% 28/30 [00:04<00:00,  5.87it/s]
2025-09-07T08:57:18.7743603Z running benchmark:  97% 29/30 [00:04<00:00,  5.85it/s]
2025-09-07T08:57:18.7743940Z running benchmark: 100% 30/30 [00:05<00:00,  5.80it/s]
2025-09-07T08:57:18.7747631Z running benchmark: 100% 30/30 [00:05<00:00,  5.83it/s]
2025-09-07T08:57:18.9049663Z 1.062x
2025-09-07T08:57:18.9054096Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:57:21.3819841Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:57:21.3820777Z   import pynvml  # type: ignore[import]
2025-09-07T08:57:23.4069873Z 
2025-09-07T08:57:25.8023170Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:57:25.8023837Z loading model: 0it [00:02, ?it/s]
2025-09-07T08:57:25.8172715Z cpu  eval  hf_T5                              
2025-09-07T08:57:50.6699880Z 
2025-09-07T08:57:50.8126914Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:57:50.9513338Z running benchmark:   3% 1/30 [00:00<00:04,  7.01it/s]
2025-09-07T08:57:51.0888662Z running benchmark:   7% 2/30 [00:00<00:03,  7.13it/s]
2025-09-07T08:57:51.2314268Z running benchmark:  10% 3/30 [00:00<00:03,  7.19it/s]
2025-09-07T08:57:51.3678393Z running benchmark:  13% 4/30 [00:00<00:03,  7.12it/s]
2025-09-07T08:57:51.5084485Z running benchmark:  17% 5/30 [00:00<00:03,  7.20it/s]
2025-09-07T08:57:51.6460237Z running benchmark:  20% 6/30 [00:00<00:03,  7.17it/s]
2025-09-07T08:57:51.7865983Z running benchmark:  23% 7/30 [00:00<00:03,  7.20it/s]
2025-09-07T08:57:51.9230537Z running benchmark:  27% 8/30 [00:01<00:03,  7.17it/s]
2025-09-07T08:57:52.0632460Z running benchmark:  30% 9/30 [00:01<00:02,  7.22it/s]
2025-09-07T08:57:52.1998117Z running benchmark:  33% 10/30 [00:01<00:02,  7.19it/s]
2025-09-07T08:57:52.3409682Z running benchmark:  37% 11/30 [00:01<00:02,  7.23it/s]
2025-09-07T08:57:52.4769621Z running benchmark:  40% 12/30 [00:01<00:02,  7.19it/s]
2025-09-07T08:57:52.6170757Z running benchmark:  43% 13/30 [00:01<00:02,  7.24it/s]
2025-09-07T08:57:52.7543551Z running benchmark:  47% 14/30 [00:01<00:02,  7.21it/s]
2025-09-07T08:57:52.8951264Z running benchmark:  50% 15/30 [00:02<00:02,  7.23it/s]
2025-09-07T08:57:53.0329443Z running benchmark:  53% 16/30 [00:02<00:01,  7.19it/s]
2025-09-07T08:57:53.1723571Z running benchmark:  57% 17/30 [00:02<00:01,  7.21it/s]
2025-09-07T08:57:53.3091958Z running benchmark:  60% 18/30 [00:02<00:01,  7.20it/s]
2025-09-07T08:57:53.4495895Z running benchmark:  63% 19/30 [00:02<00:01,  7.23it/s]
2025-09-07T08:57:53.5862190Z running benchmark:  67% 20/30 [00:02<00:01,  7.20it/s]
2025-09-07T08:57:53.7266081Z running benchmark:  70% 21/30 [00:02<00:01,  7.23it/s]
2025-09-07T08:57:53.8652288Z running benchmark:  73% 22/30 [00:03<00:01,  7.20it/s]
2025-09-07T08:57:54.0054941Z running benchmark:  77% 23/30 [00:03<00:00,  7.22it/s]
2025-09-07T08:57:54.1418897Z running benchmark:  80% 24/30 [00:03<00:00,  7.18it/s]
2025-09-07T08:57:54.2824005Z running benchmark:  83% 25/30 [00:03<00:00,  7.22it/s]
2025-09-07T08:57:54.4188501Z running benchmark:  87% 26/30 [00:03<00:00,  7.19it/s]
2025-09-07T08:57:54.5603566Z running benchmark:  90% 27/30 [00:03<00:00,  7.23it/s]
2025-09-07T08:57:54.6970917Z running benchmark:  93% 28/30 [00:03<00:00,  7.18it/s]
2025-09-07T08:57:54.8391234Z running benchmark:  97% 29/30 [00:04<00:00,  7.22it/s]
2025-09-07T08:57:54.8391986Z running benchmark: 100% 30/30 [00:04<00:00,  7.17it/s]
2025-09-07T08:57:54.8392261Z running benchmark: 100% 30/30 [00:04<00:00,  7.20it/s]
2025-09-07T08:57:54.9209347Z 1.915x
2025-09-07T08:57:54.9213515Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:57:57.2722172Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:57:57.2723057Z   import pynvml  # type: ignore[import]
2025-09-07T08:57:59.2841804Z 
2025-09-07T08:58:04.5077812Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:58:04.5078449Z loading model: 0it [00:05, ?it/s]
2025-09-07T08:58:04.5500624Z cpu  eval  hf_T5_base                         
2025-09-07T08:58:51.4248381Z 
2025-09-07T08:58:53.0437453Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T08:58:54.6554403Z running benchmark:   3% 1/30 [00:01<00:46,  1.62s/it]
2025-09-07T08:58:56.2767563Z running benchmark:   7% 2/30 [00:03<00:45,  1.61s/it]
2025-09-07T08:58:57.8841093Z running benchmark:  10% 3/30 [00:04<00:43,  1.62s/it]
2025-09-07T08:58:59.5059003Z running benchmark:  13% 4/30 [00:06<00:41,  1.61s/it]
2025-09-07T08:59:01.1222137Z running benchmark:  17% 5/30 [00:08<00:40,  1.62s/it]
2025-09-07T08:59:02.7408832Z running benchmark:  20% 6/30 [00:09<00:38,  1.62s/it]
2025-09-07T08:59:04.3410288Z running benchmark:  23% 7/30 [00:11<00:37,  1.62s/it]
2025-09-07T08:59:05.9418838Z running benchmark:  27% 8/30 [00:12<00:35,  1.61s/it]
2025-09-07T08:59:07.5506141Z running benchmark:  30% 9/30 [00:14<00:33,  1.61s/it]
2025-09-07T08:59:09.1562872Z running benchmark:  33% 10/30 [00:16<00:32,  1.61s/it]
2025-09-07T08:59:10.7631083Z running benchmark:  37% 11/30 [00:17<00:30,  1.61s/it]
2025-09-07T08:59:12.3844680Z running benchmark:  40% 12/30 [00:19<00:28,  1.61s/it]
2025-09-07T08:59:13.9925007Z running benchmark:  43% 13/30 [00:20<00:27,  1.61s/it]
2025-09-07T08:59:15.5964959Z running benchmark:  47% 14/30 [00:22<00:25,  1.61s/it]
2025-09-07T08:59:17.1951287Z running benchmark:  50% 15/30 [00:24<00:24,  1.61s/it]
2025-09-07T08:59:18.8247375Z running benchmark:  53% 16/30 [00:25<00:22,  1.61s/it]
2025-09-07T08:59:20.4334520Z running benchmark:  57% 17/30 [00:27<00:20,  1.61s/it]
2025-09-07T08:59:22.0446960Z running benchmark:  60% 18/30 [00:29<00:19,  1.61s/it]
2025-09-07T08:59:23.6447641Z running benchmark:  63% 19/30 [00:30<00:17,  1.61s/it]
2025-09-07T08:59:25.2376160Z running benchmark:  67% 20/30 [00:32<00:16,  1.61s/it]
2025-09-07T08:59:26.8398154Z running benchmark:  70% 21/30 [00:33<00:14,  1.60s/it]
2025-09-07T08:59:28.4384037Z running benchmark:  73% 22/30 [00:35<00:12,  1.60s/it]
2025-09-07T08:59:30.0455292Z running benchmark:  77% 23/30 [00:37<00:11,  1.60s/it]
2025-09-07T08:59:31.6659834Z running benchmark:  80% 24/30 [00:38<00:09,  1.60s/it]
2025-09-07T08:59:33.2721617Z running benchmark:  83% 25/30 [00:40<00:08,  1.61s/it]
2025-09-07T08:59:34.8809893Z running benchmark:  87% 26/30 [00:41<00:06,  1.61s/it]
2025-09-07T08:59:36.4758769Z running benchmark:  90% 27/30 [00:43<00:04,  1.61s/it]
2025-09-07T08:59:38.0756128Z running benchmark:  93% 28/30 [00:45<00:03,  1.60s/it]
2025-09-07T08:59:39.6800607Z running benchmark:  97% 29/30 [00:46<00:01,  1.60s/it]
2025-09-07T08:59:39.6800966Z running benchmark: 100% 30/30 [00:48<00:00,  1.60s/it]
2025-09-07T08:59:39.6801233Z running benchmark: 100% 30/30 [00:48<00:00,  1.61s/it]
2025-09-07T08:59:40.0776361Z 1.432x
2025-09-07T08:59:40.0780927Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T08:59:43.6441241Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T08:59:43.6442454Z   import pynvml  # type: ignore[import]
2025-09-07T08:59:45.6656349Z 
2025-09-07T08:59:54.9437094Z loading model: 0it [00:00, ?it/s]
2025-09-07T08:59:54.9441240Z loading model: 0it [00:09, ?it/s]
2025-09-07T08:59:55.0360651Z cpu  eval  hf_T5_large                        
2025-09-07T09:00:47.9393076Z 
2025-09-07T09:00:48.7966653Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:00:49.6517776Z running benchmark:   3% 1/30 [00:00<00:24,  1.17it/s]
2025-09-07T09:00:50.5147185Z running benchmark:   7% 2/30 [00:01<00:23,  1.17it/s]
2025-09-07T09:00:51.3705045Z running benchmark:  10% 3/30 [00:02<00:23,  1.16it/s]
2025-09-07T09:00:52.2329268Z running benchmark:  13% 4/30 [00:03<00:22,  1.17it/s]
2025-09-07T09:00:53.0918102Z running benchmark:  17% 5/30 [00:04<00:21,  1.16it/s]
2025-09-07T09:00:53.9452038Z running benchmark:  20% 6/30 [00:05<00:20,  1.16it/s]
2025-09-07T09:00:54.8057474Z running benchmark:  23% 7/30 [00:06<00:19,  1.17it/s]
2025-09-07T09:00:55.6648773Z running benchmark:  27% 8/30 [00:06<00:18,  1.16it/s]
2025-09-07T09:00:56.5296582Z running benchmark:  30% 9/30 [00:07<00:18,  1.16it/s]
2025-09-07T09:00:57.4017146Z running benchmark:  33% 10/30 [00:08<00:17,  1.16it/s]
2025-09-07T09:00:58.2913945Z running benchmark:  37% 11/30 [00:09<00:16,  1.16it/s]
2025-09-07T09:00:59.1569160Z running benchmark:  40% 12/30 [00:10<00:15,  1.15it/s]
2025-09-07T09:01:00.0151694Z running benchmark:  43% 13/30 [00:11<00:14,  1.15it/s]
2025-09-07T09:01:00.8706019Z running benchmark:  47% 14/30 [00:12<00:13,  1.15it/s]
2025-09-07T09:01:01.7309666Z running benchmark:  50% 15/30 [00:12<00:12,  1.16it/s]
2025-09-07T09:01:02.5915594Z running benchmark:  53% 16/30 [00:13<00:12,  1.16it/s]
2025-09-07T09:01:03.4517443Z running benchmark:  57% 17/30 [00:14<00:11,  1.16it/s]
2025-09-07T09:01:04.3073077Z running benchmark:  60% 18/30 [00:15<00:10,  1.16it/s]
2025-09-07T09:01:05.1714860Z running benchmark:  63% 19/30 [00:16<00:09,  1.16it/s]
2025-09-07T09:01:06.0249136Z running benchmark:  67% 20/30 [00:17<00:08,  1.16it/s]
2025-09-07T09:01:06.8857802Z running benchmark:  70% 21/30 [00:18<00:07,  1.16it/s]
2025-09-07T09:01:07.7428981Z running benchmark:  73% 22/30 [00:18<00:06,  1.16it/s]
2025-09-07T09:01:08.6057056Z running benchmark:  77% 23/30 [00:19<00:06,  1.16it/s]
2025-09-07T09:01:09.4580670Z running benchmark:  80% 24/30 [00:20<00:05,  1.16it/s]
2025-09-07T09:01:10.3177799Z running benchmark:  83% 25/30 [00:21<00:04,  1.17it/s]
2025-09-07T09:01:11.1748161Z running benchmark:  87% 26/30 [00:22<00:03,  1.17it/s]
2025-09-07T09:01:12.0339143Z running benchmark:  90% 27/30 [00:23<00:02,  1.17it/s]
2025-09-07T09:01:12.8906551Z running benchmark:  93% 28/30 [00:24<00:01,  1.17it/s]
2025-09-07T09:01:13.7523071Z running benchmark:  97% 29/30 [00:24<00:00,  1.17it/s]
2025-09-07T09:01:13.7523468Z running benchmark: 100% 30/30 [00:25<00:00,  1.16it/s]
2025-09-07T09:01:13.7523774Z running benchmark: 100% 30/30 [00:25<00:00,  1.16it/s]
2025-09-07T09:01:14.0040491Z 1.487x
2025-09-07T09:01:14.0044992Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:01:17.3506175Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:01:17.3507067Z   import pynvml  # type: ignore[import]
2025-09-07T09:01:19.3718393Z 
2025-09-07T09:01:24.0644376Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:01:24.0646362Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:01:24.0964861Z cpu  eval  hf_distil_whisper                  
2025-09-07T09:01:55.7206355Z 
2025-09-07T09:01:56.3329256Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:01:56.9380395Z running benchmark:   3% 1/30 [00:00<00:17,  1.63it/s]
2025-09-07T09:01:57.5481695Z running benchmark:   7% 2/30 [00:01<00:17,  1.64it/s]
2025-09-07T09:01:58.1717378Z running benchmark:  10% 3/30 [00:01<00:16,  1.64it/s]
2025-09-07T09:01:58.8018331Z running benchmark:  13% 4/30 [00:02<00:15,  1.63it/s]
2025-09-07T09:01:59.4210291Z running benchmark:  17% 5/30 [00:03<00:15,  1.61it/s]
2025-09-07T09:02:00.0448313Z running benchmark:  20% 6/30 [00:03<00:14,  1.61it/s]
2025-09-07T09:02:00.6691687Z running benchmark:  23% 7/30 [00:04<00:14,  1.61it/s]
2025-09-07T09:02:01.2959439Z running benchmark:  27% 8/30 [00:04<00:13,  1.61it/s]
2025-09-07T09:02:01.9273609Z running benchmark:  30% 9/30 [00:05<00:13,  1.60it/s]
2025-09-07T09:02:02.5507444Z running benchmark:  33% 10/30 [00:06<00:12,  1.60it/s]
2025-09-07T09:02:03.1773216Z running benchmark:  37% 11/30 [00:06<00:11,  1.60it/s]
2025-09-07T09:02:03.8026451Z running benchmark:  40% 12/30 [00:07<00:11,  1.60it/s]
2025-09-07T09:02:04.4297354Z running benchmark:  43% 13/30 [00:08<00:10,  1.60it/s]
2025-09-07T09:02:05.0537238Z running benchmark:  47% 14/30 [00:08<00:10,  1.60it/s]
2025-09-07T09:02:05.6762039Z running benchmark:  50% 15/30 [00:09<00:09,  1.60it/s]
2025-09-07T09:02:06.2971912Z running benchmark:  53% 16/30 [00:09<00:08,  1.60it/s]
2025-09-07T09:02:06.9242203Z running benchmark:  57% 17/30 [00:10<00:08,  1.60it/s]
2025-09-07T09:02:07.5519091Z running benchmark:  60% 18/30 [00:11<00:07,  1.60it/s]
2025-09-07T09:02:08.1781849Z running benchmark:  63% 19/30 [00:11<00:06,  1.60it/s]
2025-09-07T09:02:08.8049046Z running benchmark:  67% 20/30 [00:12<00:06,  1.60it/s]
2025-09-07T09:02:09.4086287Z running benchmark:  70% 21/30 [00:13<00:05,  1.60it/s]
2025-09-07T09:02:10.0113633Z running benchmark:  73% 22/30 [00:13<00:04,  1.61it/s]
2025-09-07T09:02:10.6802246Z running benchmark:  77% 23/30 [00:14<00:04,  1.63it/s]
2025-09-07T09:02:11.3301300Z running benchmark:  80% 24/30 [00:14<00:03,  1.59it/s]
2025-09-07T09:02:11.9453210Z running benchmark:  83% 25/30 [00:15<00:03,  1.57it/s]
2025-09-07T09:02:12.5510145Z running benchmark:  87% 26/30 [00:16<00:02,  1.59it/s]
2025-09-07T09:02:13.1602188Z running benchmark:  90% 27/30 [00:16<00:01,  1.61it/s]
2025-09-07T09:02:13.7599001Z running benchmark:  93% 28/30 [00:17<00:01,  1.62it/s]
2025-09-07T09:02:14.4370462Z running benchmark:  97% 29/30 [00:18<00:00,  1.63it/s]
2025-09-07T09:02:14.4370844Z running benchmark: 100% 30/30 [00:18<00:00,  1.58it/s]
2025-09-07T09:02:14.4371162Z running benchmark: 100% 30/30 [00:18<00:00,  1.60it/s]
2025-09-07T09:02:14.5865594Z 1.193x
2025-09-07T09:02:14.5869990Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:02:17.2960008Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:02:17.2960878Z   import pynvml  # type: ignore[import]
2025-09-07T09:02:19.3204093Z 
2025-09-07T09:02:19.5136827Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:02:19.5137143Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:02:19.5138379Z cpu  eval  lennard_jones                      
2025-09-07T09:02:26.3463092Z 
2025-09-07T09:02:26.4203619Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:02:26.4203980Z running benchmark: 100% 30/30 [00:00<00:00, 406.10it/s]
2025-09-07T09:02:26.4658426Z 0.797x
2025-09-07T09:02:26.4662141Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:02:27.9912791Z speedup             gmean=1.21x mean=1.248x
2025-09-07T09:02:27.9913247Z abs_latency         gmean=47.00x mean=135.076x
2025-09-07T09:02:27.9913917Z compilation_latency mean=22.796 seconds
2025-09-07T09:02:27.9914466Z compression_ratio   mean=0.902x
2025-09-07T09:02:27.9917359Z eager_peak_mem      gmean=205.67x mean=331.738x
2025-09-07T09:02:27.9919643Z dynamo_peak_mem     gmean=228.47x mean=350.776x
2025-09-07T09:02:27.9921725Z calls_captured      gmean=273.07x mean=611.609x
2025-09-07T09:02:27.9923876Z unique_graphs       gmean=1.66x mean=3.043x
2025-09-07T09:02:27.9925983Z graph_breaks        gmean=0.00x mean=2.174x
2025-09-07T09:02:27.9928058Z unique_graph_breaks gmean=0.00x mean=0.565x
2025-09-07T09:02:27.9930173Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T09:02:27.9932252Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T09:02:27.9934243Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T09:02:28.5718833Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *cudagraphs-true* ]]
2025-09-07T09:02:28.5719586Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *dynamic-true* ]]
2025-09-07T09:02:28.5721173Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --dynamic-shapes --dynamic-batch-only --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv
2025-09-07T09:02:28.9684159Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:02:28.9685323Z   import pynvml  # type: ignore[import]
2025-09-07T09:02:31.4279470Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:02:31.4280379Z   import pynvml  # type: ignore[import]
2025-09-07T09:02:33.4429774Z 
2025-09-07T09:02:39.0844741Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:02:39.0845247Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:02:39.1041684Z cpu  eval  dlrm                               
2025-09-07T09:02:41.8044419Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8044723Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8044941Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8045132Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8045328Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8045548Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8045740Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8045926Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8046161Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8046558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8046894Z     return mod(*inputs)
2025-09-07T09:02:41.8047188Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8047543Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8047938Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 360, in sequential_forward
2025-09-07T09:02:41.8048289Z     x = self.apply_mlp(dense_x, self.bot_l)
2025-09-07T09:02:41.8048608Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8048910Z     return layers(x)
2025-09-07T09:02:41.8049015Z 
2025-09-07T09:02:41.8049119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8049488Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8050147Z     return mod(*inputs)
2025-09-07T09:02:41.8050428Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8050753Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8051107Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8051552Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8051869Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8052209Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8052379Z 
2025-09-07T09:02:41.8052478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8052828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8053150Z     return mod(*inputs)
2025-09-07T09:02:41.8053420Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8053741Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8054087Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8054424Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8054818Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8055157Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8055323Z 
2025-09-07T09:02:41.8055448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8055794Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8056167Z     return mod(*inputs)
2025-09-07T09:02:41.8056428Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8056747Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8057089Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8057408Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8057790Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8058130Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8058285Z 
2025-09-07T09:02:41.8058388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8058727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8059030Z     return mod(*inputs)
2025-09-07T09:02:41.8059286Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8059605Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8059943Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8060265Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8060564Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8060891Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8061046Z 
2025-09-07T09:02:41.8061146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8061482Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8061786Z     return mod(*inputs)
2025-09-07T09:02:41.8062043Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8062369Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8062711Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8063057Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8063397Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8063725Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8063893Z 
2025-09-07T09:02:41.8063988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8064327Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8064680Z     return mod(*inputs)
2025-09-07T09:02:41.8064938Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8065251Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8065591Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8065923Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8066225Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8066555Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8066708Z 
2025-09-07T09:02:41.8066798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8067132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8067440Z     return mod(*inputs)
2025-09-07T09:02:41.8067735Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8068053Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8068391Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8068722Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8069058Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8069396Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8069550Z 
2025-09-07T09:02:41.8069641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8069981Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8070286Z     return mod(*inputs)
2025-09-07T09:02:41.8070541Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8070856Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8071193Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward
2025-09-07T09:02:41.8071525Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2025-09-07T09:02:41.8071825Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb
2025-09-07T09:02:41.8072152Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2025-09-07T09:02:41.8072304Z 
2025-09-07T09:02:41.8072393Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8072729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8073044Z     return mod(*inputs)
2025-09-07T09:02:41.8073297Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8073603Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8073945Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 360, in sequential_forward
2025-09-07T09:02:41.8074280Z     x = self.apply_mlp(dense_x, self.bot_l)
2025-09-07T09:02:41.8074589Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8074886Z     return layers(x)
2025-09-07T09:02:41.8074977Z 
2025-09-07T09:02:41.8075072Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8075419Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8075735Z     return mod(*inputs)
2025-09-07T09:02:41.8076032Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8076346Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8076681Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 360, in sequential_forward
2025-09-07T09:02:41.8077006Z     x = self.apply_mlp(dense_x, self.bot_l)
2025-09-07T09:02:41.8077345Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8077639Z     return layers(x)
2025-09-07T09:02:41.8077724Z 
2025-09-07T09:02:41.8077801Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8078001Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8078193Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8078391Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8078578Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8078769Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8078970Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8079163Z cudagraph partition due to non gpu ops
2025-09-07T09:02:41.8079371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8079710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8080022Z     return mod(*inputs)
2025-09-07T09:02:41.8080330Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8080658Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8081001Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T09:02:41.8081330Z     z = self.interact_features(x, ly)
2025-09-07T09:02:41.8081692Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 318, in interact_features
2025-09-07T09:02:41.8082059Z     T = torch.cat([x] + ly, dim=1).view((batch_size, -1, d))
2025-09-07T09:02:41.8082215Z 
2025-09-07T09:02:41.8082311Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8082648Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8082962Z     return mod(*inputs)
2025-09-07T09:02:41.8083218Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8083534Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8083884Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T09:02:41.8084215Z     z = self.interact_features(x, ly)
2025-09-07T09:02:41.8084538Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 320, in interact_features
2025-09-07T09:02:41.8084877Z     Z = torch.bmm(T, torch.transpose(T, 1, 2))
2025-09-07T09:02:41.8085018Z 
2025-09-07T09:02:41.8085121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8085479Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8085805Z     return mod(*inputs)
2025-09-07T09:02:41.8086086Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8086414Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8086764Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T09:02:41.8087097Z     z = self.interact_features(x, ly)
2025-09-07T09:02:41.8087413Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 337, in interact_features
2025-09-07T09:02:41.8087726Z     Zflat = Z[:, li, lj]
2025-09-07T09:02:41.8087826Z 
2025-09-07T09:02:41.8087924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8088270Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8088585Z     return mod(*inputs)
2025-09-07T09:02:41.8088912Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8089224Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8089561Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 371, in sequential_forward
2025-09-07T09:02:41.8089891Z     z = self.interact_features(x, ly)
2025-09-07T09:02:41.8090239Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 339, in interact_features
2025-09-07T09:02:41.8090559Z     R = torch.cat([x] + [Zflat], dim=1)
2025-09-07T09:02:41.8090681Z 
2025-09-07T09:02:41.8090775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8091115Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8091428Z     return mod(*inputs)
2025-09-07T09:02:41.8091688Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8091997Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8092334Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8092655Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8092956Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8093260Z     return layers(x)
2025-09-07T09:02:41.8093423Z 
2025-09-07T09:02:41.8093518Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8093859Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8094168Z     return mod(*inputs)
2025-09-07T09:02:41.8094426Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8094767Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8095110Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8095431Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8095723Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8096017Z     return layers(x)
2025-09-07T09:02:41.8096103Z 
2025-09-07T09:02:41.8096192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8096528Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8096837Z     return mod(*inputs)
2025-09-07T09:02:41.8097092Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8097399Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8097806Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8098126Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8098413Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8098703Z     return layers(x)
2025-09-07T09:02:41.8098787Z 
2025-09-07T09:02:41.8098877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8099210Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8099517Z     return mod(*inputs)
2025-09-07T09:02:41.8099777Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8100095Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8100433Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8100752Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8101043Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8101332Z     return layers(x)
2025-09-07T09:02:41.8101417Z 
2025-09-07T09:02:41.8101506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8101895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8102207Z     return mod(*inputs)
2025-09-07T09:02:41.8102460Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8102769Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8103299Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8103628Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8103923Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8104209Z     return layers(x)
2025-09-07T09:02:41.8104304Z 
2025-09-07T09:02:41.8104399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8104734Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8105045Z     return mod(*inputs)
2025-09-07T09:02:41.8105297Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8105602Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8105936Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8106258Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8106619Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8106918Z     return layers(x)
2025-09-07T09:02:41.8107017Z 
2025-09-07T09:02:41.8107112Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8107466Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8107833Z     return mod(*inputs)
2025-09-07T09:02:41.8108095Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8108413Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8108760Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8109099Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8109400Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8109694Z     return layers(x)
2025-09-07T09:02:41.8109794Z 
2025-09-07T09:02:41.8109889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:02:41.8110232Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:02:41.8110545Z     return mod(*inputs)
2025-09-07T09:02:41.8110806Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward
2025-09-07T09:02:41.8111123Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2025-09-07T09:02:41.8111460Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 375, in sequential_forward
2025-09-07T09:02:41.8111788Z     p = self.apply_mlp(z, self.top_l)
2025-09-07T09:02:41.8112080Z   File "/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 286, in apply_mlp
2025-09-07T09:02:41.8112364Z     return layers(x)
2025-09-07T09:02:41.8112460Z 
2025-09-07T09:02:52.0852759Z 
2025-09-07T09:02:52.2259798Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:02:52.3599702Z running benchmark:   7% 2/30 [00:00<00:01, 14.26it/s]
2025-09-07T09:02:52.4937937Z running benchmark:  13% 4/30 [00:00<00:01, 14.64it/s]
2025-09-07T09:02:52.6261431Z running benchmark:  20% 6/30 [00:00<00:01, 14.78it/s]
2025-09-07T09:02:52.7587206Z running benchmark:  27% 8/30 [00:00<00:01, 14.91it/s]
2025-09-07T09:02:52.8918684Z running benchmark:  33% 10/30 [00:00<00:01, 14.97it/s]
2025-09-07T09:02:53.0259226Z running benchmark:  40% 12/30 [00:00<00:01, 14.99it/s]
2025-09-07T09:02:53.1586808Z running benchmark:  47% 14/30 [00:00<00:01, 14.97it/s]
2025-09-07T09:02:53.2916411Z running benchmark:  53% 16/30 [00:01<00:00, 15.00it/s]
2025-09-07T09:02:53.4245782Z running benchmark:  60% 18/30 [00:01<00:00, 15.01it/s]
2025-09-07T09:02:53.5575287Z running benchmark:  67% 20/30 [00:01<00:00, 15.02it/s]
2025-09-07T09:02:53.6899045Z running benchmark:  73% 22/30 [00:01<00:00, 15.03it/s]
2025-09-07T09:02:53.8231003Z running benchmark:  80% 24/30 [00:01<00:00, 15.05it/s]
2025-09-07T09:02:53.9565918Z running benchmark:  87% 26/30 [00:01<00:00, 15.04it/s]
2025-09-07T09:02:54.0898054Z running benchmark:  93% 28/30 [00:01<00:00, 15.02it/s]
2025-09-07T09:02:54.0898401Z running benchmark: 100% 30/30 [00:02<00:00, 15.02it/s]
2025-09-07T09:02:54.0898674Z running benchmark: 100% 30/30 [00:02<00:00, 14.97it/s]
2025-09-07T09:02:54.1908028Z 1.030x
2025-09-07T09:02:54.1911249Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:02:56.2977930Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:02:56.2978839Z   import pynvml  # type: ignore[import]
2025-09-07T09:02:58.3082936Z 
2025-09-07T09:03:00.9435279Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T09:03:01.1804222Z 
2025-09-07T09:03:01.1804637Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:03:01.1805029Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T09:03:01.1805381Z cpu  eval  doctr_det_predictor                
2025-09-07T09:03:01.1805706Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T09:03:10.6151761Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6152341Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6152840Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6153214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6153567Z     x = module(x)
2025-09-07T09:03:10.6153660Z 
2025-09-07T09:03:10.6153792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6154267Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6154708Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6155043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6155375Z     x = module(x)
2025-09-07T09:03:10.6155461Z 
2025-09-07T09:03:10.6155560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6156021Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6156453Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6156790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6157136Z     x = module(x)
2025-09-07T09:03:10.6157449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6157806Z     out = self.conv1(x)
2025-09-07T09:03:10.6157913Z 
2025-09-07T09:03:10.6158012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6158497Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6158929Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6159256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6159681Z     x = module(x)
2025-09-07T09:03:10.6159986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6160342Z     out = self.conv2(out)
2025-09-07T09:03:10.6160449Z 
2025-09-07T09:03:10.6160546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6161098Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6161545Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6161884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6162214Z     x = module(x)
2025-09-07T09:03:10.6162516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6162855Z     out = self.conv3(out)
2025-09-07T09:03:10.6162965Z 
2025-09-07T09:03:10.6163062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6163525Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6163969Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6164359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6164694Z     x = module(x)
2025-09-07T09:03:10.6164988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6165344Z     identity = self.downsample(x)
2025-09-07T09:03:10.6165533Z 
2025-09-07T09:03:10.6165638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6166160Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6166607Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6166933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6167265Z     x = module(x)
2025-09-07T09:03:10.6167553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6167887Z     out = self.conv1(x)
2025-09-07T09:03:10.6167993Z 
2025-09-07T09:03:10.6168096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6168554Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6168986Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6169325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6169648Z     x = module(x)
2025-09-07T09:03:10.6169939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6170285Z     out = self.conv2(out)
2025-09-07T09:03:10.6170388Z 
2025-09-07T09:03:10.6170483Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6170940Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6171378Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6171718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6172051Z     x = module(x)
2025-09-07T09:03:10.6172352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6172701Z     out = self.conv3(out)
2025-09-07T09:03:10.6172819Z 
2025-09-07T09:03:10.6172918Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6173390Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6173883Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6174215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6174541Z     x = module(x)
2025-09-07T09:03:10.6174886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6175219Z     out = self.conv1(x)
2025-09-07T09:03:10.6175317Z 
2025-09-07T09:03:10.6175414Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6175883Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6176316Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6176635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6176962Z     x = module(x)
2025-09-07T09:03:10.6177245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6177673Z     out = self.conv2(out)
2025-09-07T09:03:10.6177785Z 
2025-09-07T09:03:10.6177875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6178372Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6178816Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6179130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6179453Z     x = module(x)
2025-09-07T09:03:10.6179781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6180125Z     out = self.conv3(out)
2025-09-07T09:03:10.6180227Z 
2025-09-07T09:03:10.6180317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6180770Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6181211Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6181532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6181852Z     x = module(x)
2025-09-07T09:03:10.6182133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6182470Z     out = self.conv1(x)
2025-09-07T09:03:10.6182574Z 
2025-09-07T09:03:10.6182664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6183121Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6183549Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6183860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6184187Z     x = module(x)
2025-09-07T09:03:10.6184480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6184821Z     out = self.conv2(out)
2025-09-07T09:03:10.6184919Z 
2025-09-07T09:03:10.6185009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6185461Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6185891Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6186202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6186525Z     x = module(x)
2025-09-07T09:03:10.6186809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6187186Z     out = self.conv3(out)
2025-09-07T09:03:10.6187290Z 
2025-09-07T09:03:10.6187385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6187849Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6188332Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6188645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6188968Z     x = module(x)
2025-09-07T09:03:10.6189259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6189603Z     identity = self.downsample(x)
2025-09-07T09:03:10.6189723Z 
2025-09-07T09:03:10.6189816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6190267Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6190693Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6191020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6191347Z     x = module(x)
2025-09-07T09:03:10.6191667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6192001Z     out = self.conv1(x)
2025-09-07T09:03:10.6192100Z 
2025-09-07T09:03:10.6192192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6192678Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6193108Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6193434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6193762Z     x = module(x)
2025-09-07T09:03:10.6194054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6194388Z     out = self.conv2(out)
2025-09-07T09:03:10.6194486Z 
2025-09-07T09:03:10.6194578Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6195029Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6195456Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6195770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6196103Z     x = module(x)
2025-09-07T09:03:10.6196400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6196730Z     out = self.conv3(out)
2025-09-07T09:03:10.6196827Z 
2025-09-07T09:03:10.6196922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6197374Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6197794Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6198115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6198436Z     x = module(x)
2025-09-07T09:03:10.6198727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6199056Z     out = self.conv1(x)
2025-09-07T09:03:10.6199146Z 
2025-09-07T09:03:10.6199235Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6199690Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6200215Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6200533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6200849Z     x = module(x)
2025-09-07T09:03:10.6201139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6201536Z     out = self.conv2(out)
2025-09-07T09:03:10.6201634Z 
2025-09-07T09:03:10.6201732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6202199Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6202621Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6202940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6203424Z     x = module(x)
2025-09-07T09:03:10.6203716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6204047Z     out = self.conv3(out)
2025-09-07T09:03:10.6204145Z 
2025-09-07T09:03:10.6204237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6204747Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6205193Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6205512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6205829Z     x = module(x)
2025-09-07T09:03:10.6206119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6206496Z     out = self.conv1(x)
2025-09-07T09:03:10.6206587Z 
2025-09-07T09:03:10.6206684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6207140Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6207581Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6207897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6208219Z     x = module(x)
2025-09-07T09:03:10.6208512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6208839Z     out = self.conv2(out)
2025-09-07T09:03:10.6208942Z 
2025-09-07T09:03:10.6209031Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6209489Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6209913Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6210232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6210549Z     x = module(x)
2025-09-07T09:03:10.6210837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6211164Z     out = self.conv3(out)
2025-09-07T09:03:10.6211259Z 
2025-09-07T09:03:10.6211356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6211807Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6212226Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6212544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6212864Z     x = module(x)
2025-09-07T09:03:10.6213154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6213480Z     out = self.conv1(x)
2025-09-07T09:03:10.6213629Z 
2025-09-07T09:03:10.6213719Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6214165Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6214603Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6214985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6215306Z     x = module(x)
2025-09-07T09:03:10.6215595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6215925Z     out = self.conv2(out)
2025-09-07T09:03:10.6216019Z 
2025-09-07T09:03:10.6216115Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6216568Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6216993Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6217308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6217701Z     x = module(x)
2025-09-07T09:03:10.6217991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6218357Z     out = self.conv3(out)
2025-09-07T09:03:10.6218460Z 
2025-09-07T09:03:10.6218548Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6219013Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6219441Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6219800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6220124Z     x = module(x)
2025-09-07T09:03:10.6220418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6220762Z     identity = self.downsample(x)
2025-09-07T09:03:10.6220875Z 
2025-09-07T09:03:10.6220975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6221423Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6221852Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6222167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6222490Z     x = module(x)
2025-09-07T09:03:10.6222775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6223103Z     out = self.conv1(x)
2025-09-07T09:03:10.6223198Z 
2025-09-07T09:03:10.6223286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6223740Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6224162Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6224478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6224797Z     x = module(x)
2025-09-07T09:03:10.6225086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6225416Z     out = self.conv2(out)
2025-09-07T09:03:10.6225511Z 
2025-09-07T09:03:10.6225607Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6226052Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6226482Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6226797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6227178Z     x = module(x)
2025-09-07T09:03:10.6227468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6227793Z     out = self.conv3(out)
2025-09-07T09:03:10.6227893Z 
2025-09-07T09:03:10.6227983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6228492Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6228935Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6229260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6229578Z     x = module(x)
2025-09-07T09:03:10.6229878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6230209Z     out = self.conv1(x)
2025-09-07T09:03:10.6230306Z 
2025-09-07T09:03:10.6230406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6230849Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6231277Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6231630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6231962Z     x = module(x)
2025-09-07T09:03:10.6232248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6232574Z     out = self.conv2(out)
2025-09-07T09:03:10.6232674Z 
2025-09-07T09:03:10.6232763Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6233245Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6233679Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6233993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6234319Z     x = module(x)
2025-09-07T09:03:10.6234606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6234944Z     out = self.conv3(out)
2025-09-07T09:03:10.6235042Z 
2025-09-07T09:03:10.6235139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6235590Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6236027Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6236361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6236688Z     x = module(x)
2025-09-07T09:03:10.6236991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6237318Z     out = self.conv1(x)
2025-09-07T09:03:10.6237426Z 
2025-09-07T09:03:10.6237520Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6237984Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6238416Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6238734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6239060Z     x = module(x)
2025-09-07T09:03:10.6239351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6239689Z     out = self.conv2(out)
2025-09-07T09:03:10.6239793Z 
2025-09-07T09:03:10.6239896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6240385Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6240820Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6241138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6241498Z     x = module(x)
2025-09-07T09:03:10.6241785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6242119Z     out = self.conv3(out)
2025-09-07T09:03:10.6242223Z 
2025-09-07T09:03:10.6242315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6242767Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6243199Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6243513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6243836Z     x = module(x)
2025-09-07T09:03:10.6244120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6244460Z     out = self.conv1(x)
2025-09-07T09:03:10.6244554Z 
2025-09-07T09:03:10.6244651Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6245162Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6245593Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6245912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6246234Z     x = module(x)
2025-09-07T09:03:10.6246551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6246886Z     out = self.conv2(out)
2025-09-07T09:03:10.6246991Z 
2025-09-07T09:03:10.6247080Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6247537Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6247963Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6248274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6248597Z     x = module(x)
2025-09-07T09:03:10.6248885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6249219Z     out = self.conv3(out)
2025-09-07T09:03:10.6249313Z 
2025-09-07T09:03:10.6249409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6249854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6250285Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6250601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6250926Z     x = module(x)
2025-09-07T09:03:10.6251204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6251539Z     out = self.conv1(x)
2025-09-07T09:03:10.6251638Z 
2025-09-07T09:03:10.6251729Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6252180Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6252605Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6252919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6253243Z     x = module(x)
2025-09-07T09:03:10.6253545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6253954Z     out = self.conv2(out)
2025-09-07T09:03:10.6254056Z 
2025-09-07T09:03:10.6254152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6254618Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6255084Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6255401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6255719Z     x = module(x)
2025-09-07T09:03:10.6256009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6256346Z     out = self.conv3(out)
2025-09-07T09:03:10.6256448Z 
2025-09-07T09:03:10.6256540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6257000Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6257437Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6257806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6258134Z     x = module(x)
2025-09-07T09:03:10.6260272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6260634Z     out = self.conv1(x)
2025-09-07T09:03:10.6260730Z 
2025-09-07T09:03:10.6260827Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6261327Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6261769Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6262098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6262429Z     x = module(x)
2025-09-07T09:03:10.6262719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6263053Z     out = self.conv2(out)
2025-09-07T09:03:10.6263158Z 
2025-09-07T09:03:10.6263252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6263719Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6264146Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6264464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6264790Z     x = module(x)
2025-09-07T09:03:10.6265082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6265414Z     out = self.conv3(out)
2025-09-07T09:03:10.6265512Z 
2025-09-07T09:03:10.6265601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6266054Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6266495Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6266816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6267138Z     x = module(x)
2025-09-07T09:03:10.6267425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6267765Z     identity = self.downsample(x)
2025-09-07T09:03:10.6267892Z 
2025-09-07T09:03:10.6267988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6268442Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6277273Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6277684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6278040Z     x = module(x)
2025-09-07T09:03:10.6278359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6278793Z     out = self.conv1(x)
2025-09-07T09:03:10.6278906Z 
2025-09-07T09:03:10.6279012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6279503Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6279968Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6280310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6280641Z     x = module(x)
2025-09-07T09:03:10.6280949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6281296Z     out = self.conv2(out)
2025-09-07T09:03:10.6281402Z 
2025-09-07T09:03:10.6281502Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6282020Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6282479Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6282810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6283138Z     x = module(x)
2025-09-07T09:03:10.6283430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6283807Z     out = self.conv3(out)
2025-09-07T09:03:10.6283914Z 
2025-09-07T09:03:10.6284011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6284478Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6284906Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6285230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6285559Z     x = module(x)
2025-09-07T09:03:10.6285854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6286186Z     out = self.conv1(x)
2025-09-07T09:03:10.6286280Z 
2025-09-07T09:03:10.6286371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6286831Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6287264Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6287594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6287918Z     x = module(x)
2025-09-07T09:03:10.6288206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6288537Z     out = self.conv2(out)
2025-09-07T09:03:10.6288635Z 
2025-09-07T09:03:10.6288736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6289196Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6289620Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6289942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6290263Z     x = module(x)
2025-09-07T09:03:10.6290550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6290881Z     out = self.conv3(out)
2025-09-07T09:03:10.6291020Z 
2025-09-07T09:03:10.6291110Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6291564Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6292006Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6292461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6292966Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6293479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6293993Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6294189Z 
2025-09-07T09:03:10.6294290Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6294753Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6295191Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6295639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6296138Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6296634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6297133Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6297354Z 
2025-09-07T09:03:10.6297458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6298007Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6298442Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6298859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6299382Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6299903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6300396Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6300588Z 
2025-09-07T09:03:10.6300684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6301147Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6301577Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6301990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6302469Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6302968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6303692Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6303882Z 
2025-09-07T09:03:10.6303982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6304435Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6304860Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6305267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6305902Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6306564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6307124Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6307300Z 
2025-09-07T09:03:10.6307402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6307873Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6308309Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6308736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6309214Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6309705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6310184Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6310366Z 
2025-09-07T09:03:10.6310508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6310968Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6311407Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6311857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6312343Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6312822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6313300Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6313467Z 
2025-09-07T09:03:10.6313563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6314020Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6314441Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6314850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6315318Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6315792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6316271Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6316436Z 
2025-09-07T09:03:10.6316524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6316973Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6317417Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6317541Z 
2025-09-07T09:03:10.6317635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6318083Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6318507Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6318636Z 
2025-09-07T09:03:10.6318725Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6319174Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6319655Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6319779Z 
2025-09-07T09:03:10.6319878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6320341Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6320813Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6320938Z 
2025-09-07T09:03:10.6321026Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6321478Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6321911Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6322246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6322575Z     x = module(x)
2025-09-07T09:03:10.6322668Z 
2025-09-07T09:03:10.6322758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6323208Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6323635Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6323985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6324309Z     x = module(x)
2025-09-07T09:03:10.6324390Z 
2025-09-07T09:03:10.6324486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6324988Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6325412Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6325736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6326059Z     x = module(x)
2025-09-07T09:03:10.6326137Z 
2025-09-07T09:03:10.6326234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6326678Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6327106Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6327420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6327741Z     x = module(x)
2025-09-07T09:03:10.6328037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6328364Z     out = self.conv1(x)
2025-09-07T09:03:10.6328469Z 
2025-09-07T09:03:10.6328558Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6329009Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6329437Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6329774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6330102Z     x = module(x)
2025-09-07T09:03:10.6330423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6330768Z     out = self.relu(out)
2025-09-07T09:03:10.6330871Z 
2025-09-07T09:03:10.6330976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6331438Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6331876Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6332208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6332575Z     x = module(x)
2025-09-07T09:03:10.6332877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6333209Z     out = self.conv2(out)
2025-09-07T09:03:10.6333319Z 
2025-09-07T09:03:10.6333412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6333906Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6334343Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6334664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6334983Z     x = module(x)
2025-09-07T09:03:10.6335279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6335609Z     out = self.relu(out)
2025-09-07T09:03:10.6335704Z 
2025-09-07T09:03:10.6335803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6336256Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6336689Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6337045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6337376Z     x = module(x)
2025-09-07T09:03:10.6337712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6338045Z     out = self.conv3(out)
2025-09-07T09:03:10.6338150Z 
2025-09-07T09:03:10.6338243Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6338732Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6339166Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6339486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6339802Z     x = module(x)
2025-09-07T09:03:10.6340089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6340435Z     identity = self.downsample(x)
2025-09-07T09:03:10.6340555Z 
2025-09-07T09:03:10.6340646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6341088Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6341509Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6341819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6342129Z     x = module(x)
2025-09-07T09:03:10.6342404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6342724Z     out = self.conv1(x)
2025-09-07T09:03:10.6342818Z 
2025-09-07T09:03:10.6342905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6343344Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6343768Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6344071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6344382Z     x = module(x)
2025-09-07T09:03:10.6344657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6344979Z     out = self.relu(out)
2025-09-07T09:03:10.6345074Z 
2025-09-07T09:03:10.6345162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6345597Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6346078Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6346388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6346697Z     x = module(x)
2025-09-07T09:03:10.6347019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6347346Z     out = self.conv2(out)
2025-09-07T09:03:10.6347445Z 
2025-09-07T09:03:10.6347531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6347969Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6348384Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6348702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6349022Z     x = module(x)
2025-09-07T09:03:10.6349312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6349635Z     out = self.relu(out)
2025-09-07T09:03:10.6349731Z 
2025-09-07T09:03:10.6349824Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6350309Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6350737Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6351051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6351364Z     x = module(x)
2025-09-07T09:03:10.6351676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6352003Z     out = self.conv3(out)
2025-09-07T09:03:10.6352102Z 
2025-09-07T09:03:10.6352194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6352638Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6353055Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6353362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6353676Z     x = module(x)
2025-09-07T09:03:10.6353953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6354272Z     out = self.relu(out)
2025-09-07T09:03:10.6354365Z 
2025-09-07T09:03:10.6354457Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6354908Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6355327Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6355642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6355958Z     x = module(x)
2025-09-07T09:03:10.6356239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6356563Z     out = self.conv1(x)
2025-09-07T09:03:10.6356659Z 
2025-09-07T09:03:10.6356751Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6357192Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6357614Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6357925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6358235Z     x = module(x)
2025-09-07T09:03:10.6358515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6358889Z     out = self.relu(out)
2025-09-07T09:03:10.6358984Z 
2025-09-07T09:03:10.6359074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6359516Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6359984Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6360294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6360606Z     x = module(x)
2025-09-07T09:03:10.6360965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6361287Z     out = self.conv2(out)
2025-09-07T09:03:10.6361388Z 
2025-09-07T09:03:10.6361477Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6361915Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6362338Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6362642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6362954Z     x = module(x)
2025-09-07T09:03:10.6363284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6363610Z     out = self.relu(out)
2025-09-07T09:03:10.6363703Z 
2025-09-07T09:03:10.6363789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6364231Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6364690Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6364996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6365315Z     x = module(x)
2025-09-07T09:03:10.6365589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6365909Z     out = self.conv3(out)
2025-09-07T09:03:10.6366004Z 
2025-09-07T09:03:10.6366090Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6366534Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6366949Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6367252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6367562Z     x = module(x)
2025-09-07T09:03:10.6367839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6368162Z     out = self.relu(out)
2025-09-07T09:03:10.6368255Z 
2025-09-07T09:03:10.6368345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6368790Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6369211Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6369530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6369845Z     x = module(x)
2025-09-07T09:03:10.6370122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6370442Z     out = self.conv1(x)
2025-09-07T09:03:10.6370533Z 
2025-09-07T09:03:10.6370624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6371067Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6371482Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6371850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6372161Z     x = module(x)
2025-09-07T09:03:10.6372440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6372763Z     out = self.relu(out)
2025-09-07T09:03:10.6372903Z 
2025-09-07T09:03:10.6373003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6373464Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6373893Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6374220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6374539Z     x = module(x)
2025-09-07T09:03:10.6374830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6375159Z     out = self.conv2(out)
2025-09-07T09:03:10.6375258Z 
2025-09-07T09:03:10.6375351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6375799Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6376278Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6376595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6376917Z     x = module(x)
2025-09-07T09:03:10.6377209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6377538Z     out = self.relu(out)
2025-09-07T09:03:10.6377721Z 
2025-09-07T09:03:10.6377816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6378268Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6378696Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6379016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6379327Z     x = module(x)
2025-09-07T09:03:10.6379617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6379940Z     out = self.conv3(out)
2025-09-07T09:03:10.6380039Z 
2025-09-07T09:03:10.6380131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6380574Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6380997Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6381309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6381621Z     x = module(x)
2025-09-07T09:03:10.6381901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6382234Z     identity = self.downsample(x)
2025-09-07T09:03:10.6382350Z 
2025-09-07T09:03:10.6382440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6382887Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6383307Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6383617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6383927Z     x = module(x)
2025-09-07T09:03:10.6384211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6384533Z     out = self.conv1(x)
2025-09-07T09:03:10.6384627Z 
2025-09-07T09:03:10.6384757Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6385200Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6385615Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6385926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6386280Z     x = module(x)
2025-09-07T09:03:10.6386560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6386880Z     out = self.relu(out)
2025-09-07T09:03:10.6386980Z 
2025-09-07T09:03:10.6387068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6387513Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6387931Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6388242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6388553Z     x = module(x)
2025-09-07T09:03:10.6388835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6389164Z     out = self.conv2(out)
2025-09-07T09:03:10.6389298Z 
2025-09-07T09:03:10.6389389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6389830Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6390243Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6390617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6390938Z     x = module(x)
2025-09-07T09:03:10.6391227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6391550Z     out = self.relu(out)
2025-09-07T09:03:10.6391647Z 
2025-09-07T09:03:10.6391736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6392176Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6392598Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6392907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6393215Z     x = module(x)
2025-09-07T09:03:10.6393492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6393812Z     out = self.conv3(out)
2025-09-07T09:03:10.6393906Z 
2025-09-07T09:03:10.6393994Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6394433Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6394849Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6395156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6395467Z     x = module(x)
2025-09-07T09:03:10.6395747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6396068Z     out = self.relu(out)
2025-09-07T09:03:10.6396160Z 
2025-09-07T09:03:10.6396246Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6396684Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6397099Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6397405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6397761Z     x = module(x)
2025-09-07T09:03:10.6398038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6398357Z     out = self.conv1(x)
2025-09-07T09:03:10.6398447Z 
2025-09-07T09:03:10.6398535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6399012Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6399438Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6399748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6400063Z     x = module(x)
2025-09-07T09:03:10.6400343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6400663Z     out = self.relu(out)
2025-09-07T09:03:10.6400766Z 
2025-09-07T09:03:10.6400857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6401306Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6401732Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6402129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6402462Z     x = module(x)
2025-09-07T09:03:10.6402756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6403256Z     out = self.conv2(out)
2025-09-07T09:03:10.6403354Z 
2025-09-07T09:03:10.6403450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6403971Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6404406Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6404727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6405051Z     x = module(x)
2025-09-07T09:03:10.6405338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6405663Z     out = self.relu(out)
2025-09-07T09:03:10.6405761Z 
2025-09-07T09:03:10.6405852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6406307Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6406735Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6407047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6407369Z     x = module(x)
2025-09-07T09:03:10.6407656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6407988Z     out = self.conv3(out)
2025-09-07T09:03:10.6408083Z 
2025-09-07T09:03:10.6408181Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6408623Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6409048Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6409355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6409670Z     x = module(x)
2025-09-07T09:03:10.6409951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6410269Z     out = self.relu(out)
2025-09-07T09:03:10.6410365Z 
2025-09-07T09:03:10.6410452Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6410897Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6411394Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6411717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6412036Z     x = module(x)
2025-09-07T09:03:10.6412383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6412713Z     out = self.conv1(x)
2025-09-07T09:03:10.6412810Z 
2025-09-07T09:03:10.6412905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6413353Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6413781Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6414099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6414414Z     x = module(x)
2025-09-07T09:03:10.6414694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6415018Z     out = self.relu(out)
2025-09-07T09:03:10.6415117Z 
2025-09-07T09:03:10.6415207Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6415705Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6416134Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6416441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6416765Z     x = module(x)
2025-09-07T09:03:10.6417080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6417413Z     out = self.conv2(out)
2025-09-07T09:03:10.6417513Z 
2025-09-07T09:03:10.6417654Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6418096Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6418520Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6418833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6419151Z     x = module(x)
2025-09-07T09:03:10.6419427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6419751Z     out = self.relu(out)
2025-09-07T09:03:10.6419847Z 
2025-09-07T09:03:10.6419932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6420376Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6420806Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6421111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6421429Z     x = module(x)
2025-09-07T09:03:10.6421707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6422037Z     out = self.conv3(out)
2025-09-07T09:03:10.6422129Z 
2025-09-07T09:03:10.6422219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6422662Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6423092Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6423401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6423718Z     x = module(x)
2025-09-07T09:03:10.6423993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6424367Z     out = self.relu(out)
2025-09-07T09:03:10.6424461Z 
2025-09-07T09:03:10.6424548Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6424996Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6425469Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6425776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6426093Z     x = module(x)
2025-09-07T09:03:10.6426371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6426695Z     out = self.conv1(x)
2025-09-07T09:03:10.6426789Z 
2025-09-07T09:03:10.6426876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6427321Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6427741Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6428054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6428368Z     x = module(x)
2025-09-07T09:03:10.6428679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6429011Z     out = self.relu(out)
2025-09-07T09:03:10.6429104Z 
2025-09-07T09:03:10.6429193Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6429631Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6430084Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6430389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6430702Z     x = module(x)
2025-09-07T09:03:10.6430981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6431304Z     out = self.conv2(out)
2025-09-07T09:03:10.6431395Z 
2025-09-07T09:03:10.6431481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6431923Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6432339Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6432652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6432976Z     x = module(x)
2025-09-07T09:03:10.6433255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6433581Z     out = self.relu(out)
2025-09-07T09:03:10.6433671Z 
2025-09-07T09:03:10.6433759Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6434208Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6434623Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6434927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6435256Z     x = module(x)
2025-09-07T09:03:10.6435534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6435855Z     out = self.conv3(out)
2025-09-07T09:03:10.6435945Z 
2025-09-07T09:03:10.6436032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6436471Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6436901Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6437248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6437567Z     x = module(x)
2025-09-07T09:03:10.6437841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6438228Z     identity = self.downsample(x)
2025-09-07T09:03:10.6438352Z 
2025-09-07T09:03:10.6438439Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6438877Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6439293Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6439604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6439923Z     x = module(x)
2025-09-07T09:03:10.6440198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6440525Z     out = self.conv1(x)
2025-09-07T09:03:10.6440616Z 
2025-09-07T09:03:10.6440703Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6441137Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6441616Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6441923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6442246Z     x = module(x)
2025-09-07T09:03:10.6442520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6442845Z     out = self.relu(out)
2025-09-07T09:03:10.6442975Z 
2025-09-07T09:03:10.6443064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6443502Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6443918Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6444231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6444544Z     x = module(x)
2025-09-07T09:03:10.6444830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6445149Z     out = self.conv2(out)
2025-09-07T09:03:10.6445239Z 
2025-09-07T09:03:10.6445324Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6445765Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6446187Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6446494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6446804Z     x = module(x)
2025-09-07T09:03:10.6447082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6447401Z     out = self.relu(out)
2025-09-07T09:03:10.6447492Z 
2025-09-07T09:03:10.6447585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6448031Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6448446Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6448755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6449071Z     x = module(x)
2025-09-07T09:03:10.6449353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6449675Z     out = self.conv3(out)
2025-09-07T09:03:10.6449767Z 
2025-09-07T09:03:10.6449914Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6450363Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6450783Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6451091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6451446Z     x = module(x)
2025-09-07T09:03:10.6451724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6452049Z     out = self.relu(out)
2025-09-07T09:03:10.6452138Z 
2025-09-07T09:03:10.6452228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6452671Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6453086Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6453396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6453709Z     x = module(x)
2025-09-07T09:03:10.6453986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6454310Z     out = self.conv1(x)
2025-09-07T09:03:10.6454434Z 
2025-09-07T09:03:10.6454521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6454962Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6455380Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6455724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6456042Z     x = module(x)
2025-09-07T09:03:10.6456325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6456649Z     out = self.relu(out)
2025-09-07T09:03:10.6456739Z 
2025-09-07T09:03:10.6456828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6457272Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6457726Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6458037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6458353Z     x = module(x)
2025-09-07T09:03:10.6458634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6458953Z     out = self.conv2(out)
2025-09-07T09:03:10.6459053Z 
2025-09-07T09:03:10.6459139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6459585Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6460012Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6460323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6460633Z     x = module(x)
2025-09-07T09:03:10.6460916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6461239Z     out = self.relu(out)
2025-09-07T09:03:10.6461328Z 
2025-09-07T09:03:10.6461419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6461876Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6462300Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6462615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6463007Z     x = module(x)
2025-09-07T09:03:10.6463295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6463618Z     out = self.conv3(out)
2025-09-07T09:03:10.6463718Z 
2025-09-07T09:03:10.6463806Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6464305Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6464734Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6465047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6465364Z     x = module(x)
2025-09-07T09:03:10.6465653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6465983Z     out = self.relu(out)
2025-09-07T09:03:10.6466073Z 
2025-09-07T09:03:10.6466173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6466617Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6467041Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6467397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6467727Z     x = module(x)
2025-09-07T09:03:10.6468013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6468335Z     out = self.conv1(x)
2025-09-07T09:03:10.6468433Z 
2025-09-07T09:03:10.6468521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6469009Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6469441Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6469760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6470074Z     x = module(x)
2025-09-07T09:03:10.6470363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6470696Z     out = self.relu(out)
2025-09-07T09:03:10.6470791Z 
2025-09-07T09:03:10.6470888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6471333Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6471771Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6472091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6472413Z     x = module(x)
2025-09-07T09:03:10.6472704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6473029Z     out = self.conv2(out)
2025-09-07T09:03:10.6473135Z 
2025-09-07T09:03:10.6473223Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6473673Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6474114Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6474430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6474750Z     x = module(x)
2025-09-07T09:03:10.6475036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6475364Z     out = self.relu(out)
2025-09-07T09:03:10.6475455Z 
2025-09-07T09:03:10.6475550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6475994Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6476505Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6476833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6477164Z     x = module(x)
2025-09-07T09:03:10.6477500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6477834Z     out = self.conv3(out)
2025-09-07T09:03:10.6477940Z 
2025-09-07T09:03:10.6478033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6478495Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6478934Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6479251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6479573Z     x = module(x)
2025-09-07T09:03:10.6479865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6480196Z     out = self.relu(out)
2025-09-07T09:03:10.6480293Z 
2025-09-07T09:03:10.6480390Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6480875Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6481306Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6481630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6481958Z     x = module(x)
2025-09-07T09:03:10.6482276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6482611Z     out = self.conv1(x)
2025-09-07T09:03:10.6482712Z 
2025-09-07T09:03:10.6482807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6483263Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6483692Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6484005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6484326Z     x = module(x)
2025-09-07T09:03:10.6484617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6484950Z     out = self.relu(out)
2025-09-07T09:03:10.6485051Z 
2025-09-07T09:03:10.6485148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6485595Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6486024Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6486341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6486664Z     x = module(x)
2025-09-07T09:03:10.6486945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6487278Z     out = self.conv2(out)
2025-09-07T09:03:10.6487386Z 
2025-09-07T09:03:10.6487475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6487927Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6488351Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6488664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6488989Z     x = module(x)
2025-09-07T09:03:10.6489278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6489652Z     out = self.relu(out)
2025-09-07T09:03:10.6489744Z 
2025-09-07T09:03:10.6489842Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6490285Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6490752Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6491070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6491393Z     x = module(x)
2025-09-07T09:03:10.6491674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6492004Z     out = self.conv3(out)
2025-09-07T09:03:10.6492108Z 
2025-09-07T09:03:10.6492198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6492646Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6493078Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6493386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6493706Z     x = module(x)
2025-09-07T09:03:10.6494033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6494366Z     out = self.relu(out)
2025-09-07T09:03:10.6494457Z 
2025-09-07T09:03:10.6494545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6494997Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6495457Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6495778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6496101Z     x = module(x)
2025-09-07T09:03:10.6496383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6496716Z     out = self.conv1(x)
2025-09-07T09:03:10.6496814Z 
2025-09-07T09:03:10.6496902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6497357Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6497871Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6498183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6498510Z     x = module(x)
2025-09-07T09:03:10.6498797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6499130Z     out = self.relu(out)
2025-09-07T09:03:10.6499219Z 
2025-09-07T09:03:10.6499312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6499758Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6500185Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6500507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6500828Z     x = module(x)
2025-09-07T09:03:10.6501108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6501440Z     out = self.conv2(out)
2025-09-07T09:03:10.6501541Z 
2025-09-07T09:03:10.6501631Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6502081Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6502506Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6502858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6503354Z     x = module(x)
2025-09-07T09:03:10.6503642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6504067Z     out = self.relu(out)
2025-09-07T09:03:10.6504164Z 
2025-09-07T09:03:10.6504257Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6504706Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6505131Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6505451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6505773Z     x = module(x)
2025-09-07T09:03:10.6506054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6506385Z     out = self.conv3(out)
2025-09-07T09:03:10.6506488Z 
2025-09-07T09:03:10.6506578Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6507024Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6507496Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6507813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6508136Z     x = module(x)
2025-09-07T09:03:10.6508426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6508758Z     out = self.relu(out)
2025-09-07T09:03:10.6508898Z 
2025-09-07T09:03:10.6508992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6509439Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6509868Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6510188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6510512Z     x = module(x)
2025-09-07T09:03:10.6510798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6511128Z     out = self.conv1(x)
2025-09-07T09:03:10.6511218Z 
2025-09-07T09:03:10.6511318Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6511770Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6512192Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6512512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6512836Z     x = module(x)
2025-09-07T09:03:10.6513123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6513451Z     out = self.relu(out)
2025-09-07T09:03:10.6513544Z 
2025-09-07T09:03:10.6513634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6514089Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6514515Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6514834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6515152Z     x = module(x)
2025-09-07T09:03:10.6515446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6515776Z     out = self.conv2(out)
2025-09-07T09:03:10.6515870Z 
2025-09-07T09:03:10.6516022Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6516327Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6516395Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6516599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6516688Z     x = module(x)
2025-09-07T09:03:10.6516899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6516957Z     out = self.relu(out)
2025-09-07T09:03:10.6516960Z 
2025-09-07T09:03:10.6517060Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6517361Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6517434Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6517638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6517697Z     x = module(x)
2025-09-07T09:03:10.6517910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6517970Z     out = self.conv3(out)
2025-09-07T09:03:10.6517975Z 
2025-09-07T09:03:10.6518111Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6518433Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6518496Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6518755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6518809Z     x = module(x)
2025-09-07T09:03:10.6519024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 158, in forward
2025-09-07T09:03:10.6519097Z     identity = self.downsample(x)
2025-09-07T09:03:10.6519100Z 
2025-09-07T09:03:10.6519198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6519499Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6519571Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6519778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6519829Z     x = module(x)
2025-09-07T09:03:10.6520042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6520101Z     out = self.conv1(x)
2025-09-07T09:03:10.6520106Z 
2025-09-07T09:03:10.6520195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6520500Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6520567Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6520776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6520828Z     x = module(x)
2025-09-07T09:03:10.6521034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6521100Z     out = self.relu(out)
2025-09-07T09:03:10.6521103Z 
2025-09-07T09:03:10.6521191Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6521500Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6521563Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6521769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6521860Z     x = module(x)
2025-09-07T09:03:10.6522067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6522134Z     out = self.conv2(out)
2025-09-07T09:03:10.6522136Z 
2025-09-07T09:03:10.6522226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6522569Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6522632Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6522836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6522903Z     x = module(x)
2025-09-07T09:03:10.6523107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6523167Z     out = self.relu(out)
2025-09-07T09:03:10.6523170Z 
2025-09-07T09:03:10.6523261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6523561Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6523622Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6523860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6523915Z     x = module(x)
2025-09-07T09:03:10.6524119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6524177Z     out = self.conv3(out)
2025-09-07T09:03:10.6524179Z 
2025-09-07T09:03:10.6524265Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6524595Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6524658Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6524877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6524938Z     x = module(x)
2025-09-07T09:03:10.6525144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6525209Z     out = self.relu(out)
2025-09-07T09:03:10.6525213Z 
2025-09-07T09:03:10.6525304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6525602Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6525671Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6525872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6525933Z     x = module(x)
2025-09-07T09:03:10.6526136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 146, in forward
2025-09-07T09:03:10.6526194Z     out = self.conv1(x)
2025-09-07T09:03:10.6526197Z 
2025-09-07T09:03:10.6526293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6526593Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6526668Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6526869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6526923Z     x = module(x)
2025-09-07T09:03:10.6527136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 148, in forward
2025-09-07T09:03:10.6527193Z     out = self.relu(out)
2025-09-07T09:03:10.6527197Z 
2025-09-07T09:03:10.6527298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6527594Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6527695Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6527897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6534206Z     x = module(x)
2025-09-07T09:03:10.6534578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 150, in forward
2025-09-07T09:03:10.6534645Z     out = self.conv2(out)
2025-09-07T09:03:10.6534649Z 
2025-09-07T09:03:10.6534762Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6535095Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6535172Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6535395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6535452Z     x = module(x)
2025-09-07T09:03:10.6535669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 152, in forward
2025-09-07T09:03:10.6535729Z     out = self.relu(out)
2025-09-07T09:03:10.6535732Z 
2025-09-07T09:03:10.6535834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6536188Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6536253Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6536461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6536512Z     x = module(x)
2025-09-07T09:03:10.6536760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 154, in forward
2025-09-07T09:03:10.6536822Z     out = self.conv3(out)
2025-09-07T09:03:10.6536825Z 
2025-09-07T09:03:10.6536922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6537224Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 190, in forward
2025-09-07T09:03:10.6537283Z     feats = self.feat_extractor(x)
2025-09-07T09:03:10.6537491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/_utils.py", line 69, in forward
2025-09-07T09:03:10.6537545Z     x = module(x)
2025-09-07T09:03:10.6537829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 161, in forward
2025-09-07T09:03:10.6537887Z     out = self.relu(out)
2025-09-07T09:03:10.6537891Z 
2025-09-07T09:03:10.6537981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6538285Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6538351Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6538665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6538805Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6539126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6539259Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6539262Z 
2025-09-07T09:03:10.6539353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6539674Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6539740Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6540060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6540236Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6540561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6540726Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6540729Z 
2025-09-07T09:03:10.6540825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6541138Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6541200Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6541504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6541630Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6541935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6542061Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6542067Z 
2025-09-07T09:03:10.6542190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6542499Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6542565Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6542902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 86, in forward
2025-09-07T09:03:10.6542980Z     out.append(self.upsample(out[-1]) + t)
2025-09-07T09:03:10.6542983Z 
2025-09-07T09:03:10.6543074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6543375Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6543435Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6543732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6543854Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6544161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6544286Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6544289Z 
2025-09-07T09:03:10.6544375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6544675Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6544735Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6545032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 86, in forward
2025-09-07T09:03:10.6545104Z     out.append(self.upsample(out[-1]) + t)
2025-09-07T09:03:10.6545107Z 
2025-09-07T09:03:10.6545196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6545491Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6545551Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6545853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in forward
2025-09-07T09:03:10.6545975Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6546330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 83, in <listcomp>
2025-09-07T09:03:10.6546450Z     _x: List[torch.Tensor] = [branch(t) for branch, t in zip(self.in_branches, x)]
2025-09-07T09:03:10.6546495Z 
2025-09-07T09:03:10.6546588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6546883Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6546941Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6547243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 86, in forward
2025-09-07T09:03:10.6547311Z     out.append(self.upsample(out[-1]) + t)
2025-09-07T09:03:10.6547314Z 
2025-09-07T09:03:10.6547402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6547702Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6547763Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6548112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6548252Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6548558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6548664Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6548668Z 
2025-09-07T09:03:10.6548796Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6549102Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6549169Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6549467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6549574Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6549882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6549985Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6549989Z 
2025-09-07T09:03:10.6550078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6550371Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6550433Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6550726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6550830Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6551140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6551246Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6551250Z 
2025-09-07T09:03:10.6551341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6551638Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6551706Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6551997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6552156Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6552464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6552603Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6552607Z 
2025-09-07T09:03:10.6552700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6552993Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6553056Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6553353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6553459Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6553767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6553872Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6553876Z 
2025-09-07T09:03:10.6554001Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6554300Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6554363Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6554655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6554795Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6555112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6555229Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6555233Z 
2025-09-07T09:03:10.6555332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6555634Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6555705Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6556002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6556115Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6556426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6556532Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6556535Z 
2025-09-07T09:03:10.6556630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6556926Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6556992Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6557286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in forward
2025-09-07T09:03:10.6557394Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6557699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 89, in <listcomp>
2025-09-07T09:03:10.6560414Z     out = [branch(t) for branch, t in zip(self.out_branches, out[::-1])]
2025-09-07T09:03:10.6560420Z 
2025-09-07T09:03:10.6560573Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6560880Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 193, in forward
2025-09-07T09:03:10.6560951Z     feat_concat = self.fpn(feats)
2025-09-07T09:03:10.6561254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 91, in forward
2025-09-07T09:03:10.6561364Z     return torch.cat(out, dim=1)
2025-09-07T09:03:10.6561367Z 
2025-09-07T09:03:10.6561443Z cudagraph partition due to non gpu ops
2025-09-07T09:03:10.6561534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6561844Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6561918Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6561921Z 
2025-09-07T09:03:10.6562017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6562318Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6562386Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6562392Z 
2025-09-07T09:03:10.6562487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6562827Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6562900Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6562903Z 
2025-09-07T09:03:10.6562992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6563327Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 194, in forward
2025-09-07T09:03:10.6563393Z     logits = self.prob_head(feat_concat)
2025-09-07T09:03:10.6563398Z 
2025-09-07T09:03:10.6563469Z cudagraph partition due to non gpu ops
2025-09-07T09:03:10.6563564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:10.6563860Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 202, in forward
2025-09-07T09:03:10.6563953Z     prob_map = _bf16_to_float32(torch.sigmoid(logits))
2025-09-07T09:03:10.6564197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/utils/pytorch.py", line 30, in _bf16_to_float32
2025-09-07T09:03:10.6564298Z     return x.float() if x.dtype == torch.bfloat16 else x
2025-09-07T09:03:10.6564302Z 
2025-09-07T09:03:38.8860138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:38.8860666Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/detection/core.py", line 95, in __call__
2025-09-07T09:03:38.8861117Z     for bmap in (proba_map >= self.bin_thresh).astype(np.uint8)
2025-09-07T09:03:38.8861305Z 
2025-09-07T09:03:39.9283827Z 
2025-09-07T09:03:40.1450039Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:03:40.3477488Z running benchmark:   3% 1/30 [00:00<00:06,  4.62it/s]
2025-09-07T09:03:40.5490525Z running benchmark:   7% 2/30 [00:00<00:05,  4.80it/s]
2025-09-07T09:03:40.7509905Z running benchmark:  10% 3/30 [00:00<00:05,  4.88it/s]
2025-09-07T09:03:40.9462188Z running benchmark:  13% 4/30 [00:00<00:05,  4.90it/s]
2025-09-07T09:03:41.1483571Z running benchmark:  17% 5/30 [00:01<00:05,  4.98it/s]
2025-09-07T09:03:41.3513137Z running benchmark:  20% 6/30 [00:01<00:04,  4.97it/s]
2025-09-07T09:03:41.5531001Z running benchmark:  23% 7/30 [00:01<00:04,  4.96it/s]
2025-09-07T09:03:41.7548496Z running benchmark:  27% 8/30 [00:01<00:04,  4.96it/s]
2025-09-07T09:03:41.9549532Z running benchmark:  30% 9/30 [00:01<00:04,  4.96it/s]
2025-09-07T09:03:42.1458630Z running benchmark:  33% 10/30 [00:02<00:04,  4.97it/s]
2025-09-07T09:03:42.3406288Z running benchmark:  37% 11/30 [00:02<00:03,  5.05it/s]
2025-09-07T09:03:42.5435927Z running benchmark:  40% 12/30 [00:02<00:03,  5.07it/s]
2025-09-07T09:03:42.7392582Z running benchmark:  43% 13/30 [00:02<00:03,  5.03it/s]
2025-09-07T09:03:42.9430741Z running benchmark:  47% 14/30 [00:02<00:03,  5.05it/s]
2025-09-07T09:03:43.1460155Z running benchmark:  50% 15/30 [00:03<00:02,  5.01it/s]
2025-09-07T09:03:43.3488850Z running benchmark:  53% 16/30 [00:03<00:02,  4.98it/s]
2025-09-07T09:03:43.5430144Z running benchmark:  57% 17/30 [00:03<00:02,  4.97it/s]
2025-09-07T09:03:43.7461539Z running benchmark:  60% 18/30 [00:03<00:02,  5.02it/s]
2025-09-07T09:03:43.9472197Z running benchmark:  63% 19/30 [00:03<00:02,  4.99it/s]
2025-09-07T09:03:44.1507539Z running benchmark:  67% 20/30 [00:04<00:02,  4.99it/s]
2025-09-07T09:03:44.3524174Z running benchmark:  70% 21/30 [00:04<00:01,  4.96it/s]
2025-09-07T09:03:44.5482297Z running benchmark:  73% 22/30 [00:04<00:01,  4.96it/s]
2025-09-07T09:03:44.7426132Z running benchmark:  77% 23/30 [00:04<00:01,  5.01it/s]
2025-09-07T09:03:44.9388435Z running benchmark:  80% 24/30 [00:04<00:01,  5.05it/s]
2025-09-07T09:03:45.1411955Z running benchmark:  83% 25/30 [00:05<00:00,  5.06it/s]
2025-09-07T09:03:45.3448890Z running benchmark:  87% 26/30 [00:05<00:00,  5.02it/s]
2025-09-07T09:03:45.5468038Z running benchmark:  90% 27/30 [00:05<00:00,  4.99it/s]
2025-09-07T09:03:45.7411799Z running benchmark:  93% 28/30 [00:05<00:00,  4.98it/s]
2025-09-07T09:03:45.9390211Z running benchmark:  97% 29/30 [00:05<00:00,  5.03it/s]
2025-09-07T09:03:45.9390820Z running benchmark: 100% 30/30 [00:06<00:00,  5.03it/s]
2025-09-07T09:03:45.9391896Z running benchmark: 100% 30/30 [00:06<00:00,  4.99it/s]
2025-09-07T09:03:46.0641952Z 2.164x
2025-09-07T09:03:46.0646328Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:03:48.5741551Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:03:48.5742551Z   import pynvml  # type: ignore[import]
2025-09-07T09:03:50.5886056Z 
2025-09-07T09:03:53.1754993Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T09:03:53.2091433Z 
2025-09-07T09:03:53.2091697Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:03:53.2093806Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T09:03:53.2094227Z cpu  eval  doctr_reco_predictor               
2025-09-07T09:03:53.2094607Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T09:03:56.9339659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9340198Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9340646Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9340781Z 
2025-09-07T09:03:56.9340896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9341348Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9341740Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9341898Z 
2025-09-07T09:03:56.9342003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9342432Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9342812Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9342945Z 
2025-09-07T09:03:56.9343043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9343439Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9344135Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9344259Z 
2025-09-07T09:03:56.9344351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9344742Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9345123Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9345333Z 
2025-09-07T09:03:56.9345433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9345819Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9346196Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9346326Z 
2025-09-07T09:03:56.9346416Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9346815Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9347197Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9347316Z 
2025-09-07T09:03:56.9347409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9347811Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9348186Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9348306Z 
2025-09-07T09:03:56.9348488Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9348886Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9349250Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9349373Z 
2025-09-07T09:03:56.9349461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9349921Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9350307Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9350431Z 
2025-09-07T09:03:56.9350532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9350922Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9351297Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9351426Z 
2025-09-07T09:03:56.9351525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9351923Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9352295Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9352426Z 
2025-09-07T09:03:56.9352524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9352939Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9353322Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9353449Z 
2025-09-07T09:03:56.9353550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9353940Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9354319Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9354444Z 
2025-09-07T09:03:56.9354533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9354928Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9355322Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9355444Z 
2025-09-07T09:03:56.9355539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9355948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9356326Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9356445Z 
2025-09-07T09:03:56.9356588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9356982Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9357347Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9357469Z 
2025-09-07T09:03:56.9357562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9357997Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9358372Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9358492Z 
2025-09-07T09:03:56.9358595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9358990Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9359367Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9359494Z 
2025-09-07T09:03:56.9359585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9359978Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9360355Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9360474Z 
2025-09-07T09:03:56.9360566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9361061Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9361492Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9361625Z 
2025-09-07T09:03:56.9361732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9362176Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9362566Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9362698Z 
2025-09-07T09:03:56.9362803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9363242Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9363627Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9363751Z 
2025-09-07T09:03:56.9363851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9364264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9364645Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9364763Z 
2025-09-07T09:03:56.9364866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9365264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9365633Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9365754Z 
2025-09-07T09:03:56.9365844Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9366244Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9366625Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9366740Z 
2025-09-07T09:03:56.9366837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9367225Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9367610Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9367732Z 
2025-09-07T09:03:56.9367822Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9368217Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9368594Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9368715Z 
2025-09-07T09:03:56.9368805Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9369199Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9369631Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9369750Z 
2025-09-07T09:03:56.9369848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9370237Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9370656Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9370781Z 
2025-09-07T09:03:56.9370889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9371277Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9371647Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9371765Z 
2025-09-07T09:03:56.9371861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9372250Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9372619Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9372742Z 
2025-09-07T09:03:56.9372832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9373224Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9373633Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9373752Z 
2025-09-07T09:03:56.9373844Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9374233Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9374607Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9374787Z 
2025-09-07T09:03:56.9374886Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9375285Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9375653Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9375776Z 
2025-09-07T09:03:56.9375867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9376258Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9376636Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9376754Z 
2025-09-07T09:03:56.9376851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9377254Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9377681Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9377808Z 
2025-09-07T09:03:56.9377901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9378295Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9378670Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9378788Z 
2025-09-07T09:03:56.9378878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9379267Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9379644Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9379762Z 
2025-09-07T09:03:56.9379857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9380250Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9380617Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9380739Z 
2025-09-07T09:03:56.9380830Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9381224Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9381650Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9381766Z 
2025-09-07T09:03:56.9381857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9382251Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9382739Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9382857Z 
2025-09-07T09:03:56.9382956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9383354Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9383718Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9383842Z 
2025-09-07T09:03:56.9383933Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9384329Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9384706Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9384823Z 
2025-09-07T09:03:56.9384919Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9385304Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9385677Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9385802Z 
2025-09-07T09:03:56.9385929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9386319Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9386691Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9386807Z 
2025-09-07T09:03:56.9386896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9387321Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9387701Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9387820Z 
2025-09-07T09:03:56.9387917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9388321Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9388697Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9388824Z 
2025-09-07T09:03:56.9388915Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9389310Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9389680Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9389797Z 
2025-09-07T09:03:56.9389889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9390279Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9390652Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9390770Z 
2025-09-07T09:03:56.9390868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9391260Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9391629Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9391756Z 
2025-09-07T09:03:56.9391851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9392245Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9392621Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9392740Z 
2025-09-07T09:03:56.9392838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9393225Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9393596Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9393762Z 
2025-09-07T09:03:56.9393851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9394240Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9394618Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9394732Z 
2025-09-07T09:03:56.9394859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9395251Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9395621Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9395738Z 
2025-09-07T09:03:56.9395836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9396231Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9396614Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9396739Z 
2025-09-07T09:03:56.9396832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9397221Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9397599Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9397715Z 
2025-09-07T09:03:56.9397805Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:03:56.9398244Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 206, in forward
2025-09-07T09:03:56.9398626Z     features = self.feat_extractor(x)
2025-09-07T09:03:56.9398742Z 
2025-09-07T09:04:09.2372708Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:09.2373677Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/doctr/models/recognition/crnn/pytorch.py", line 75, in ctc_best_path
2025-09-07T09:04:09.2374192Z     probs = F.softmax(logits, dim=-1).max(dim=-1).values.min(dim=1).values
2025-09-07T09:04:09.2374406Z 
2025-09-07T09:04:10.1680456Z 
2025-09-07T09:04:10.2789454Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:04:10.3871746Z running benchmark:  20% 6/30 [00:00<00:00, 54.21it/s]
2025-09-07T09:04:10.4950135Z running benchmark:  40% 12/30 [00:00<00:00, 54.94it/s]
2025-09-07T09:04:10.6023913Z running benchmark:  60% 18/30 [00:00<00:00, 55.29it/s]
2025-09-07T09:04:10.7113856Z running benchmark:  80% 24/30 [00:00<00:00, 55.52it/s]
2025-09-07T09:04:10.7114191Z running benchmark: 100% 30/30 [00:00<00:00, 55.40it/s]
2025-09-07T09:04:10.7114461Z running benchmark: 100% 30/30 [00:00<00:00, 55.27it/s]
2025-09-07T09:04:10.7851056Z 1.483x
2025-09-07T09:04:10.7854842Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:04:12.9987795Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:04:12.9988943Z   import pynvml  # type: ignore[import]
2025-09-07T09:04:15.0131076Z 
2025-09-07T09:04:15.0657401Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T09:04:15.0658134Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T09:04:15.0658759Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T09:04:15.0659358Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T09:04:15.9104596Z 
2025-09-07T09:04:15.9105153Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:04:15.9115887Z cpu  eval  drq                                
2025-09-07T09:04:18.1885661Z cudagraph partition due to non gpu ops
2025-09-07T09:04:18.1886304Z cudagraph partition due to non gpu ops
2025-09-07T09:04:18.1887461Z cudagraph partition due to non gpu ops
2025-09-07T09:04:18.1887749Z cudagraph partition due to non gpu ops
2025-09-07T09:04:18.1888012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1888412Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1888921Z     return mod(*inputs)
2025-09-07T09:04:18.1889189Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1889498Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1889796Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T09:04:18.1890064Z     h = self.forward_conv(obs)
2025-09-07T09:04:18.1890344Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 43, in forward_conv
2025-09-07T09:04:18.1890638Z     conv = torch.relu(self.convs[0](obs))
2025-09-07T09:04:18.1890777Z 
2025-09-07T09:04:18.1890888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1891253Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1891571Z     return mod(*inputs)
2025-09-07T09:04:18.1891817Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1892199Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1892481Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T09:04:18.1892738Z     h = self.forward_conv(obs)
2025-09-07T09:04:18.1893007Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T09:04:18.1893299Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T09:04:18.1893504Z 
2025-09-07T09:04:18.1893611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1893960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1894274Z     return mod(*inputs)
2025-09-07T09:04:18.1894500Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1894776Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1895047Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T09:04:18.1895309Z     h = self.forward_conv(obs)
2025-09-07T09:04:18.1895568Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T09:04:18.1895859Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T09:04:18.1895994Z 
2025-09-07T09:04:18.1896099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1896451Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1896765Z     return mod(*inputs)
2025-09-07T09:04:18.1897002Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1897293Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1897568Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T09:04:18.1897894Z     h = self.forward_conv(obs)
2025-09-07T09:04:18.1898155Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T09:04:18.1898439Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T09:04:18.1898568Z 
2025-09-07T09:04:18.1898670Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1899006Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1899322Z     return mod(*inputs)
2025-09-07T09:04:18.1899550Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1899836Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1900105Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 56, in forward
2025-09-07T09:04:18.1900357Z     h = self.forward_conv(obs)
2025-09-07T09:04:18.1900662Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 47, in forward_conv
2025-09-07T09:04:18.1900948Z     conv = torch.relu(self.convs[i](conv))
2025-09-07T09:04:18.1901071Z 
2025-09-07T09:04:18.1901172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1901508Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1901871Z     return mod(*inputs)
2025-09-07T09:04:18.1902104Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1902385Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1902657Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 61, in forward
2025-09-07T09:04:18.1902904Z     out = self.head(h)
2025-09-07T09:04:18.1903009Z 
2025-09-07T09:04:18.1903299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1903683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1904012Z     return mod(*inputs)
2025-09-07T09:04:18.1904250Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 97, in forward
2025-09-07T09:04:18.1904544Z     obs = self.encoder(obs, detach=detach_encoder)
2025-09-07T09:04:18.1904820Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 63, in forward
2025-09-07T09:04:18.1905144Z     out = torch.tanh(out)
2025-09-07T09:04:18.1905251Z 
2025-09-07T09:04:18.1905352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1905702Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1906015Z     return mod(*inputs)
2025-09-07T09:04:18.1906248Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T09:04:18.1906606Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T09:04:18.1906755Z 
2025-09-07T09:04:18.1906863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1907233Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1907559Z     return mod(*inputs)
2025-09-07T09:04:18.1907822Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T09:04:18.1908109Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T09:04:18.1908258Z 
2025-09-07T09:04:18.1908361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1908719Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1909039Z     return mod(*inputs)
2025-09-07T09:04:18.1909273Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T09:04:18.1909548Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T09:04:18.1909687Z 
2025-09-07T09:04:18.1909790Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1910126Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1910441Z     return mod(*inputs)
2025-09-07T09:04:18.1910669Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T09:04:18.1910932Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T09:04:18.1911071Z 
2025-09-07T09:04:18.1911162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1911505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1911815Z     return mod(*inputs)
2025-09-07T09:04:18.1912031Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 99, in forward
2025-09-07T09:04:18.1912302Z     mu, log_std = self.trunk(obs).chunk(2, dim=-1)
2025-09-07T09:04:18.1912441Z 
2025-09-07T09:04:18.1912536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:18.1912876Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:18.1913247Z     return mod(*inputs)
2025-09-07T09:04:18.1913476Z   File "/torchbench/torchbenchmark/models/drq/drq.py", line 105, in forward
2025-09-07T09:04:18.1913736Z     std = log_std.exp()
2025-09-07T09:04:18.1913829Z 
2025-09-07T09:04:28.2444670Z 
2025-09-07T09:04:28.3456332Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:04:28.4122063Z running benchmark:  60% 18/30 [00:00<00:00, 178.45it/s]
2025-09-07T09:04:28.4123077Z running benchmark: 100% 30/30 [00:00<00:00, 179.51it/s]
2025-09-07T09:04:28.4662114Z 0.726x
2025-09-07T09:04:28.4665480Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:04:30.5097082Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:04:30.5098022Z   import pynvml  # type: ignore[import]
2025-09-07T09:04:32.5199344Z 
2025-09-07T09:04:33.6230986Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:04:33.6231288Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:04:33.6394791Z cpu  eval  fastNLP_Bert                       
2025-09-07T09:04:34.3586000Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T09:04:34.3586682Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T09:04:34.3587194Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T09:04:34.3587797Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T09:04:34.3588385Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T09:04:34.3588876Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:04:34.3589322Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T09:04:34.3589995Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:04:34.3590637Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(*inputs)
2025-09-07T09:04:34.3591294Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T09:04:34.3591976Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     sequence_output = self.bert(words)
2025-09-07T09:04:34.3592689Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T09:04:34.3593375Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.model(words)
2025-09-07T09:04:34.3594073Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T09:04:34.3595338Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T09:04:34.3595942Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:04:34.3596316Z W0907 09:04:34.357813 161043 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:04:36.1049861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:36.1050372Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T09:04:36.1051312Z     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T09:04:36.1051815Z 
2025-09-07T09:04:36.1051928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:36.1052360Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 441, in forward
2025-09-07T09:04:36.1052745Z     seq_len = word_mask.sum(dim=-1)
2025-09-07T09:04:36.1052873Z 
2025-09-07T09:04:44.8283408Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8283903Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8284332Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8284725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8285133Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8285860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8286294Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8286707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8287091Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8287556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8287934Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8288086Z 
2025-09-07T09:04:44.8288191Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8288576Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8288958Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8289329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8289735Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8290122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8290531Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8290938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8291328Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8291705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8292059Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8292201Z 
2025-09-07T09:04:44.8292300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8292707Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8293072Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8293474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8293856Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8294243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8294646Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8295134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8295519Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8295901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8296383Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8296530Z 
2025-09-07T09:04:44.8296617Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8296829Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8297045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8297480Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8297858Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8298239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8298651Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8299047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8299456Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8299912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8300313Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8300692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8301094Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8301237Z 
2025-09-07T09:04:44.8301337Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8301718Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8302090Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8302535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8302923Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8303469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8303892Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8304287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8304641Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8304783Z 
2025-09-07T09:04:44.8304884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8305265Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8305637Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8306000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8306382Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8306770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8307167Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8307557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8307944Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8308316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8308754Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8308916Z 
2025-09-07T09:04:44.8309014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8309392Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8309761Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8310183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8310567Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8310948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8311356Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8311753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8312113Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8312254Z 
2025-09-07T09:04:44.8312353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8312731Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8313108Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8313518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8313919Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8314317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8314784Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8315193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8315575Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8315947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8316334Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8316480Z 
2025-09-07T09:04:44.8316584Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8316956Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8317312Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8317671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8318062Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8318446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8318851Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8319247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8319630Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8320003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8320358Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8320490Z 
2025-09-07T09:04:44.8320582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8320950Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8321317Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8321676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8322105Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8322479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8322880Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8323320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8323692Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8324061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8324414Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8324561Z 
2025-09-07T09:04:44.8324640Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8324850Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8325070Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8325459Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8325840Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8326209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8326646Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8327034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8327437Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8328979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8329392Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8329777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8330141Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8330281Z 
2025-09-07T09:04:44.8330378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8330759Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8331132Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8331497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8331874Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8332260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8332653Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8333034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8333388Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8333520Z 
2025-09-07T09:04:44.8333616Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8333985Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8334354Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8334713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8335093Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8335465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8335851Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8336249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8336737Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8337117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8337531Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8337730Z 
2025-09-07T09:04:44.8337827Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8338195Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8338562Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8338913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8339294Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8339676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8340081Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8340488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8340845Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8341028Z 
2025-09-07T09:04:44.8341126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8341502Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8341871Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8342274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8342659Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8343045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8343446Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8343845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8344247Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8344624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8344999Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8345152Z 
2025-09-07T09:04:44.8345253Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8345634Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8346000Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8346370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8346761Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8347149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8347558Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8347952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8348352Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8348726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8349089Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8349225Z 
2025-09-07T09:04:44.8349326Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8349761Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8350129Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8350493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8350925Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8351309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8351706Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8352099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8352476Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8352849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8353206Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8353355Z 
2025-09-07T09:04:44.8353436Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8353639Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8353860Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8354272Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8354634Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8354994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8363869Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8364407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8364857Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8365284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8365683Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8366075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8366444Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8366582Z 
2025-09-07T09:04:44.8366693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8367075Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8367444Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8367818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8368217Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8368610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8369008Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8369393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8369755Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8369897Z 
2025-09-07T09:04:44.8369997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8370387Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8370775Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8371146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8371534Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8371970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8372364Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8372745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8373176Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8373555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8373930Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8374083Z 
2025-09-07T09:04:44.8374195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8374574Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8374944Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8375305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8375693Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8376111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8376517Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8376920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8377277Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8377479Z 
2025-09-07T09:04:44.8377615Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8377988Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8378351Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8378710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8379095Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8379474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8379872Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8380271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8380649Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8381026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8381391Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8381537Z 
2025-09-07T09:04:44.8381630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8382002Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8382366Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8382730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8383116Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8383485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8383883Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8384279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8384654Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8385071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8385424Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8385565Z 
2025-09-07T09:04:44.8385658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8386031Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8386440Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8386792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8387179Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8387556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8387960Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8388355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8388725Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8389093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8389492Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8389635Z 
2025-09-07T09:04:44.8389723Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8389928Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8390144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8390551Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8390923Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8391281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8391659Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8392040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8392438Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8392841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8393230Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8393601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8393956Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8394096Z 
2025-09-07T09:04:44.8394189Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8394557Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8394922Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8395273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8395655Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8396043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8396438Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8396821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8397170Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8397310Z 
2025-09-07T09:04:44.8397405Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8397772Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8398187Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8398543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8398925Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8399346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8399734Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8400118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8400503Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8400890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8401256Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8401405Z 
2025-09-07T09:04:44.8401506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8401878Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8402240Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8402637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8403025Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8403604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8404102Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8404512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8404884Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8405028Z 
2025-09-07T09:04:44.8405127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8405538Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8405914Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8406282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8406674Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8407057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8407469Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8407863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8408241Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8408615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8408975Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8409118Z 
2025-09-07T09:04:44.8409224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8409588Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8409954Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8410311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8410698Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8411076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8411542Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8411939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8412313Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8412686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8413119Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8413254Z 
2025-09-07T09:04:44.8413346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8413717Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8414086Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8414442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8414820Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8415196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8415594Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8416049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8416446Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8416816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8417179Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8417385Z 
2025-09-07T09:04:44.8417500Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8417705Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8417922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8418291Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8418655Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8419017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8419407Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8419780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8420184Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8420580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8420971Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8421351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8421699Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8421835Z 
2025-09-07T09:04:44.8421928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8422299Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8422664Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8423022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8423397Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8423774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8424165Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8424548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8424949Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8425078Z 
2025-09-07T09:04:44.8425172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8425548Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8425949Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8426292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8426662Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8427031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8427405Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8427771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8428141Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8428506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8428867Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8429051Z 
2025-09-07T09:04:44.8429146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8429505Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8429856Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8430246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8430639Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8431008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8431407Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8431799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8432147Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8432281Z 
2025-09-07T09:04:44.8432383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8432754Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8433110Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8433493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8433893Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8434272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8434677Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8435061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8435430Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8435790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8436175Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8436315Z 
2025-09-07T09:04:44.8436408Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8436765Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8437156Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8437506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8437927Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8438307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8438743Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8439149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8439515Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8439873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8440222Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8440354Z 
2025-09-07T09:04:44.8440444Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8440804Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8441166Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8441518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8441924Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8442301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8442692Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8443100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8443528Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8443897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8444258Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8444402Z 
2025-09-07T09:04:44.8444477Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8444678Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8444887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8445255Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8445613Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8445968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8446345Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8446714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8447113Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8447502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8447891Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8448266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8448611Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8448742Z 
2025-09-07T09:04:44.8448847Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8449209Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8449562Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8449933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8450312Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8450735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8451114Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8451489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8451878Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8452008Z 
2025-09-07T09:04:44.8452097Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8452479Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8452830Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8453181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8453547Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8453916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8454291Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8454662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8455085Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8455449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8455809Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8455954Z 
2025-09-07T09:04:44.8456044Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8456445Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8456801Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8457150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8457601Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8457971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8458368Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8458755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8459099Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8459229Z 
2025-09-07T09:04:44.8459320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8459675Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8460029Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8460372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8460740Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8461106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8461499Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8461882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8462243Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8462609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8462965Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8463102Z 
2025-09-07T09:04:44.8463259Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8463617Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8463966Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8464316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8464729Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8465097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8465482Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8465868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8466232Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8466588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8466929Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8467056Z 
2025-09-07T09:04:44.8467147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8467541Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8467903Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8468249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8468620Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8469023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8469413Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8469797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8470167Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8470521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8470873Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8471011Z 
2025-09-07T09:04:44.8471084Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8471275Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8471482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8471834Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8472189Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8472533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8472905Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8473271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8473655Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8474040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8474419Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8474788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8475131Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8475263Z 
2025-09-07T09:04:44.8475352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8475712Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8476142Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8476492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8476858Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8477272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8477651Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8478022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8478365Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8478492Z 
2025-09-07T09:04:44.8478582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8478934Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8479291Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8479636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8480006Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8480409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8480790Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8481163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8481532Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8481933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8482295Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8482443Z 
2025-09-07T09:04:44.8482534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8482894Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8483245Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8483594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8483962Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8484332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8484724Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8485109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8485452Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8485581Z 
2025-09-07T09:04:44.8485670Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8486028Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8486397Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8486891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8487283Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8487679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8488083Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8488474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8488893Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8489260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8489618Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8489763Z 
2025-09-07T09:04:44.8489862Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8490275Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8490635Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8491004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8491394Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8491779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8492182Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8492570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8492953Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8493364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8493739Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8493877Z 
2025-09-07T09:04:44.8493980Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8494349Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8494754Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8495118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8495506Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8495884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8496278Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8496681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8497057Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8497457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8497817Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8497964Z 
2025-09-07T09:04:44.8498043Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8498247Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8498470Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8498840Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8499199Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8499567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8499958Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8500342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8500740Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8501128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8501515Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8501897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8502305Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8502437Z 
2025-09-07T09:04:44.8502533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8502905Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8503459Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8503831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8504219Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8504604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8505005Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8505394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8505758Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8505891Z 
2025-09-07T09:04:44.8505995Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8506364Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8507101Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8507470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8507873Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8508334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8508733Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8509127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8509515Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8509892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8510262Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8510427Z 
2025-09-07T09:04:44.8510528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8510898Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8511262Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8511626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8512003Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8512383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8512790Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8513191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8513552Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8513686Z 
2025-09-07T09:04:44.8513781Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8514152Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8514515Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8514877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8515262Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8515635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8516109Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8516507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8516884Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8517311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8517687Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8517838Z 
2025-09-07T09:04:44.8517933Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8518305Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8518672Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8519025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8519406Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8519788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8520189Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8520626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8520999Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8521367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8521753Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8521887Z 
2025-09-07T09:04:44.8521986Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8522355Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8522714Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8523071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8523452Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8523832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8524231Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8524619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8524999Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8525366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8525728Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8525869Z 
2025-09-07T09:04:44.8525949Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8526157Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8526377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8526745Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8527108Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8527460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8527844Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8528223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8528618Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8529054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8529436Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8529819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8530223Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8530353Z 
2025-09-07T09:04:44.8530453Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8530816Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8531178Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8531538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8531921Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8532303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8532687Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8533068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8533462Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8533592Z 
2025-09-07T09:04:44.8533689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8534058Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8534416Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8534813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8535203Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8535584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8535969Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8536348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8536736Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8537118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8537538Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8537688Z 
2025-09-07T09:04:44.8537787Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8538163Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8538532Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8538894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8539278Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8539652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8540061Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8540464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8540838Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8540969Z 
2025-09-07T09:04:44.8541066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8541427Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8541789Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8542192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8542575Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8542954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8543402Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8543804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8544180Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8544554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8544909Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8545058Z 
2025-09-07T09:04:44.8545152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8545525Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8545890Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8546297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8546681Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8547060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8547461Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8547905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8548282Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8548641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8548996Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8549132Z 
2025-09-07T09:04:44.8549224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8549592Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8549958Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8550308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8550690Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8551069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8551464Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8551855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8552224Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8552592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8552959Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8553097Z 
2025-09-07T09:04:44.8553183Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8553378Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8553595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8553963Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8554330Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8554684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8555120Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8555497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8555895Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8556344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8556724Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8557105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8557458Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8557589Z 
2025-09-07T09:04:44.8557699Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8558065Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8558427Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8558780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8559161Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8559609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8560002Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8560378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8560736Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8560874Z 
2025-09-07T09:04:44.8561006Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8561375Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8561738Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8562084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8562462Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8562840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8563231Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8563607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8563986Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8564357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8564724Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8564874Z 
2025-09-07T09:04:44.8564976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8565336Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8565699Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8566063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8566443Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8566820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8567220Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8567621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8567979Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8568171Z 
2025-09-07T09:04:44.8568273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8568646Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8569006Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8569415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8569807Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8570198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8570602Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8570997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8571379Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8571754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8572113Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8572253Z 
2025-09-07T09:04:44.8572347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8572771Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8573147Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8573508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8573927Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8574302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8574700Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8575096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8575465Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8575834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8576187Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8576325Z 
2025-09-07T09:04:44.8576474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8576834Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8577199Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8577597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8577980Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8578359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8578750Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8579150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8579542Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8579921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8580293Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8580439Z 
2025-09-07T09:04:44.8580520Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8580725Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8580954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8581444Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8581812Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8582186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8582620Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8583014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8583420Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8583816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8584214Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8584601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8584968Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8585102Z 
2025-09-07T09:04:44.8585208Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8585575Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8585982Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8586349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8586740Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8587125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8587546Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8587936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8588298Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8588429Z 
2025-09-07T09:04:44.8588530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8588896Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8589269Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8589633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8590014Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8590398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8590779Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8591167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8591555Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8591931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8592300Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8592450Z 
2025-09-07T09:04:44.8592547Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8592913Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8593281Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8593642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8594028Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8594409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8594860Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8595264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8595621Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8595793Z 
2025-09-07T09:04:44.8595890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8596261Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8596626Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8596986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8597370Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8597740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8598146Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8598543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8598922Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8599330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 271, in forward
2025-09-07T09:04:44.8599696Z     mixed_query_layer = self.query(hidden_states)
2025-09-07T09:04:44.8599849Z 
2025-09-07T09:04:44.8599945Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8600349Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8600726Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8601084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8601465Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8601848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8602254Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8602655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8603029Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8603573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 272, in forward
2025-09-07T09:04:44.8603932Z     mixed_key_layer = self.key(hidden_states)
2025-09-07T09:04:44.8604065Z 
2025-09-07T09:04:44.8604167Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8604540Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8604900Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8605259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8605640Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8606022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8606421Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8606808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 320, in forward
2025-09-07T09:04:44.8607185Z     self_output = self.self(input_tensor, attention_mask)
2025-09-07T09:04:44.8607551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 273, in forward
2025-09-07T09:04:44.8608026Z     mixed_value_layer = self.value(hidden_states)
2025-09-07T09:04:44.8608169Z 
2025-09-07T09:04:44.8608254Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8608455Z cudagraph partition due to non gpu ops
2025-09-07T09:04:44.8608676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8609054Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8609494Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8609852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8610243Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8610632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 362, in forward
2025-09-07T09:04:44.8611034Z     attention_output = self.attention(hidden_states, attention_mask)
2025-09-07T09:04:44.8611437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 321, in forward
2025-09-07T09:04:44.8611820Z     attention_output = self.output(self_output, input_tensor)
2025-09-07T09:04:44.8612202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 307, in forward
2025-09-07T09:04:44.8612613Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8612749Z 
2025-09-07T09:04:44.8612851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8613223Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8613585Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8613998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8614387Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8614772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8615154Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8615540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 335, in forward
2025-09-07T09:04:44.8615901Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8616030Z 
2025-09-07T09:04:44.8616130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8616502Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8616860Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8617222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8617665Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8618050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 363, in forward
2025-09-07T09:04:44.8618447Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:04:44.8618830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 336, in forward
2025-09-07T09:04:44.8619220Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:04:44.8619605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 150, in gelu
2025-09-07T09:04:44.8619977Z     return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
2025-09-07T09:04:44.8620124Z 
2025-09-07T09:04:44.8620229Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8620594Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 513, in forward
2025-09-07T09:04:44.8620961Z     encoded_layers = self.encoder(embedding_output,
2025-09-07T09:04:44.8621360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 384, in forward
2025-09-07T09:04:44.8621753Z     hidden_states = layer_module(hidden_states, attention_mask)
2025-09-07T09:04:44.8622133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 364, in forward
2025-09-07T09:04:44.8622569Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:04:44.8622966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 348, in forward
2025-09-07T09:04:44.8623315Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:04:44.8623444Z 
2025-09-07T09:04:44.8623538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8630024Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 518, in forward
2025-09-07T09:04:44.8630494Z     pooled_output = self.pooler(sequence_output)
2025-09-07T09:04:44.8630905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 404, in forward
2025-09-07T09:04:44.8631315Z     pooled_output = self.dense(first_token_tensor)
2025-09-07T09:04:44.8631470Z 
2025-09-07T09:04:44.8631580Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:44.8632030Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 518, in forward
2025-09-07T09:04:44.8632412Z     pooled_output = self.pooler(sequence_output)
2025-09-07T09:04:44.8632771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/modules/encoder/bert.py", line 405, in forward
2025-09-07T09:04:44.8633142Z     pooled_output = self.activation(pooled_output)
2025-09-07T09:04:44.8633286Z 
2025-09-07T09:04:48.8835161Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:48.8835743Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 140, in torch_dynamo_resume_in_forward_at_137
2025-09-07T09:04:48.8836249Z     return self.dropout(outputs)
2025-09-07T09:04:48.8836630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/embedding.py", line 160, in dropout
2025-09-07T09:04:48.8837015Z     return self.dropout_layer(words)
2025-09-07T09:04:48.8837162Z 
2025-09-07T09:04:49.5021260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:04:49.5021730Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 266, in torch_dynamo_resume_in_forward_at_265
2025-09-07T09:04:49.5022220Z     logits = self.qa_outputs(sequence_output)  # [batch_size, seq_len, num_labels]
2025-09-07T09:04:49.5022440Z 
2025-09-07T09:04:49.7049065Z 
2025-09-07T09:04:49.8188254Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:04:49.9296597Z running benchmark:   3% 1/30 [00:00<00:03,  8.80it/s]
2025-09-07T09:04:50.0414622Z running benchmark:   7% 2/30 [00:00<00:03,  8.93it/s]
2025-09-07T09:04:50.1531645Z running benchmark:  10% 3/30 [00:00<00:03,  8.94it/s]
2025-09-07T09:04:50.2644497Z running benchmark:  13% 4/30 [00:00<00:02,  8.94it/s]
2025-09-07T09:04:50.3766006Z running benchmark:  17% 5/30 [00:00<00:02,  8.96it/s]
2025-09-07T09:04:50.4897639Z running benchmark:  20% 6/30 [00:00<00:02,  8.94it/s]
2025-09-07T09:04:50.6000870Z running benchmark:  23% 7/30 [00:00<00:02,  8.91it/s]
2025-09-07T09:04:50.7126014Z running benchmark:  27% 8/30 [00:00<00:02,  8.96it/s]
2025-09-07T09:04:50.8234931Z running benchmark:  30% 9/30 [00:01<00:02,  8.94it/s]
2025-09-07T09:04:50.9347989Z running benchmark:  33% 10/30 [00:01<00:02,  8.96it/s]
2025-09-07T09:04:51.0466331Z running benchmark:  37% 11/30 [00:01<00:02,  8.97it/s]
2025-09-07T09:04:51.1570641Z running benchmark:  40% 12/30 [00:01<00:02,  8.96it/s]
2025-09-07T09:04:51.2694650Z running benchmark:  43% 13/30 [00:01<00:01,  8.99it/s]
2025-09-07T09:04:51.3795197Z running benchmark:  47% 14/30 [00:01<00:01,  8.96it/s]
2025-09-07T09:04:51.4908914Z running benchmark:  50% 15/30 [00:01<00:01,  9.00it/s]
2025-09-07T09:04:51.6029080Z running benchmark:  53% 16/30 [00:01<00:01,  8.99it/s]
2025-09-07T09:04:51.7140663Z running benchmark:  57% 17/30 [00:01<00:01,  8.97it/s]
2025-09-07T09:04:51.8238058Z running benchmark:  60% 18/30 [00:02<00:01,  8.98it/s]
2025-09-07T09:04:51.9348275Z running benchmark:  63% 19/30 [00:02<00:01,  9.02it/s]
2025-09-07T09:04:52.0452755Z running benchmark:  67% 20/30 [00:02<00:01,  9.02it/s]
2025-09-07T09:04:52.1577109Z running benchmark:  70% 21/30 [00:02<00:00,  9.03it/s]
2025-09-07T09:04:52.2697242Z running benchmark:  73% 22/30 [00:02<00:00,  8.99it/s]
2025-09-07T09:04:52.3816862Z running benchmark:  77% 23/30 [00:02<00:00,  8.97it/s]
2025-09-07T09:04:52.4929805Z running benchmark:  80% 24/30 [00:02<00:00,  8.98it/s]
2025-09-07T09:04:52.6032008Z running benchmark:  83% 25/30 [00:02<00:00,  8.96it/s]
2025-09-07T09:04:52.7152862Z running benchmark:  87% 26/30 [00:02<00:00,  8.99it/s]
2025-09-07T09:04:52.8266904Z running benchmark:  90% 27/30 [00:03<00:00,  8.97it/s]
2025-09-07T09:04:52.9382856Z running benchmark:  93% 28/30 [00:03<00:00,  8.97it/s]
2025-09-07T09:04:53.0494318Z running benchmark:  97% 29/30 [00:03<00:00,  8.97it/s]
2025-09-07T09:04:53.0494658Z running benchmark: 100% 30/30 [00:03<00:00,  8.99it/s]
2025-09-07T09:04:53.0494940Z running benchmark: 100% 30/30 [00:03<00:00,  8.97it/s]
2025-09-07T09:04:53.1315339Z 1.618x
2025-09-07T09:04:53.1318734Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:04:55.3301688Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:04:55.3302650Z   import pynvml  # type: ignore[import]
2025-09-07T09:04:57.3513609Z 
2025-09-07T09:04:57.7612867Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:04:57.7613183Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:04:57.7641950Z cpu  eval  functorch_dp_cifar10               
2025-09-07T09:05:01.5440662Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5440969Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5441177Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5441414Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5441656Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5441871Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5442063Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5442252Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5442446Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5442636Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5442830Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5443028Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5443223Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5443430Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5443623Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5443814Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5444013Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5444213Z cudagraph partition due to non gpu ops
2025-09-07T09:05:01.5444449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5444841Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5445187Z     return mod(*inputs)
2025-09-07T09:05:01.5445544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5445907Z     return self._forward_impl(x)
2025-09-07T09:05:01.5446261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 268, in _forward_impl
2025-09-07T09:05:01.5446641Z     x = self.conv1(x)
2025-09-07T09:05:01.5447091Z 
2025-09-07T09:05:01.5447229Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5447596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5447929Z     return mod(*inputs)
2025-09-07T09:05:01.5448250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5448697Z     return self._forward_impl(x)
2025-09-07T09:05:01.5449051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 270, in _forward_impl
2025-09-07T09:05:01.5449402Z     x = self.relu(x)
2025-09-07T09:05:01.5449503Z 
2025-09-07T09:05:01.5449599Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5449952Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5450271Z     return mod(*inputs)
2025-09-07T09:05:01.5450571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5450933Z     return self._forward_impl(x)
2025-09-07T09:05:01.5451275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 271, in _forward_impl
2025-09-07T09:05:01.5451626Z     x = self.maxpool(x)
2025-09-07T09:05:01.5451719Z 
2025-09-07T09:05:01.5451903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5452245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5452560Z     return mod(*inputs)
2025-09-07T09:05:01.5452859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5453197Z     return self._forward_impl(x)
2025-09-07T09:05:01.5453590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5453942Z     x = self.layer1(x)
2025-09-07T09:05:01.5454252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5454584Z     out = self.conv1(x)
2025-09-07T09:05:01.5454674Z 
2025-09-07T09:05:01.5454778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5455121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5455430Z     return mod(*inputs)
2025-09-07T09:05:01.5455739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5456078Z     return self._forward_impl(x)
2025-09-07T09:05:01.5456416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5456766Z     x = self.layer1(x)
2025-09-07T09:05:01.5457070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5457406Z     out = self.relu(out)
2025-09-07T09:05:01.5457507Z 
2025-09-07T09:05:01.5457610Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5458019Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5458333Z     return mod(*inputs)
2025-09-07T09:05:01.5458650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5458989Z     return self._forward_impl(x)
2025-09-07T09:05:01.5459324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5459673Z     x = self.layer1(x)
2025-09-07T09:05:01.5459972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5460309Z     out = self.conv2(out)
2025-09-07T09:05:01.5460410Z 
2025-09-07T09:05:01.5460508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5460898Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5461207Z     return mod(*inputs)
2025-09-07T09:05:01.5461507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5461891Z     return self._forward_impl(x)
2025-09-07T09:05:01.5462225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5462575Z     x = self.layer1(x)
2025-09-07T09:05:01.5462870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5463204Z     out = self.relu(out)
2025-09-07T09:05:01.5463301Z 
2025-09-07T09:05:01.5463399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5463753Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5464064Z     return mod(*inputs)
2025-09-07T09:05:01.5464364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5464704Z     return self._forward_impl(x)
2025-09-07T09:05:01.5465034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5465445Z     x = self.layer1(x)
2025-09-07T09:05:01.5465753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5466078Z     out = self.conv1(x)
2025-09-07T09:05:01.5466176Z 
2025-09-07T09:05:01.5466273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5466654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5466974Z     return mod(*inputs)
2025-09-07T09:05:01.5467274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5467616Z     return self._forward_impl(x)
2025-09-07T09:05:01.5467951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5468298Z     x = self.layer1(x)
2025-09-07T09:05:01.5468600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5468926Z     out = self.relu(out)
2025-09-07T09:05:01.5469034Z 
2025-09-07T09:05:01.5469125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5469463Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5469774Z     return mod(*inputs)
2025-09-07T09:05:01.5470068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5470400Z     return self._forward_impl(x)
2025-09-07T09:05:01.5470738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5471080Z     x = self.layer1(x)
2025-09-07T09:05:01.5471377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5471712Z     out = self.conv2(out)
2025-09-07T09:05:01.5471821Z 
2025-09-07T09:05:01.5471912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5472244Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5472552Z     return mod(*inputs)
2025-09-07T09:05:01.5472851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5473189Z     return self._forward_impl(x)
2025-09-07T09:05:01.5473520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 273, in _forward_impl
2025-09-07T09:05:01.5473919Z     x = self.layer1(x)
2025-09-07T09:05:01.5474219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5474545Z     out = self.relu(out)
2025-09-07T09:05:01.5474650Z 
2025-09-07T09:05:01.5474739Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5475138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5475463Z     return mod(*inputs)
2025-09-07T09:05:01.5475769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5476112Z     return self._forward_impl(x)
2025-09-07T09:05:01.5476476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5476833Z     x = self.layer2(x)
2025-09-07T09:05:01.5477144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5477473Z     out = self.conv1(x)
2025-09-07T09:05:01.5477571Z 
2025-09-07T09:05:01.5477668Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5478014Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5478329Z     return mod(*inputs)
2025-09-07T09:05:01.5478658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5479001Z     return self._forward_impl(x)
2025-09-07T09:05:01.5479337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5479689Z     x = self.layer2(x)
2025-09-07T09:05:01.5480018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5480350Z     out = self.relu(out)
2025-09-07T09:05:01.5480457Z 
2025-09-07T09:05:01.5480551Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5480893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5481203Z     return mod(*inputs)
2025-09-07T09:05:01.5481496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5481836Z     return self._forward_impl(x)
2025-09-07T09:05:01.5482169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5482513Z     x = self.layer2(x)
2025-09-07T09:05:01.5482802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5483129Z     out = self.conv2(out)
2025-09-07T09:05:01.5483241Z 
2025-09-07T09:05:01.5483335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5483678Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5483989Z     return mod(*inputs)
2025-09-07T09:05:01.5484285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5484623Z     return self._forward_impl(x)
2025-09-07T09:05:01.5484960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5485307Z     x = self.layer2(x)
2025-09-07T09:05:01.5485631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 100, in forward
2025-09-07T09:05:01.5485975Z     identity = self.downsample(x)
2025-09-07T09:05:01.5486105Z 
2025-09-07T09:05:01.5486204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5486558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5486881Z     return mod(*inputs)
2025-09-07T09:05:01.5487254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5487603Z     return self._forward_impl(x)
2025-09-07T09:05:01.5487951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5488304Z     x = self.layer2(x)
2025-09-07T09:05:01.5488655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5488984Z     out = self.relu(out)
2025-09-07T09:05:01.5489091Z 
2025-09-07T09:05:01.5489186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5489532Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5489847Z     return mod(*inputs)
2025-09-07T09:05:01.5490145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5490487Z     return self._forward_impl(x)
2025-09-07T09:05:01.5490825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5491172Z     x = self.layer2(x)
2025-09-07T09:05:01.5491471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5491869Z     out = self.conv1(x)
2025-09-07T09:05:01.5491971Z 
2025-09-07T09:05:01.5492061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5492401Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5492710Z     return mod(*inputs)
2025-09-07T09:05:01.5493057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5493401Z     return self._forward_impl(x)
2025-09-07T09:05:01.5493735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5494088Z     x = self.layer2(x)
2025-09-07T09:05:01.5494383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5494703Z     out = self.relu(out)
2025-09-07T09:05:01.5494805Z 
2025-09-07T09:05:01.5494897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5495236Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5495644Z     return mod(*inputs)
2025-09-07T09:05:01.5495938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5496274Z     return self._forward_impl(x)
2025-09-07T09:05:01.5496608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5496956Z     x = self.layer2(x)
2025-09-07T09:05:01.5497251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5497576Z     out = self.conv2(out)
2025-09-07T09:05:01.5497712Z 
2025-09-07T09:05:01.5497809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5498153Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5498469Z     return mod(*inputs)
2025-09-07T09:05:01.5498810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5499159Z     return self._forward_impl(x)
2025-09-07T09:05:01.5499507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 274, in _forward_impl
2025-09-07T09:05:01.5499855Z     x = self.layer2(x)
2025-09-07T09:05:01.5500166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5500554Z     out = self.relu(out)
2025-09-07T09:05:01.5500659Z 
2025-09-07T09:05:01.5500765Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5501117Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5501438Z     return mod(*inputs)
2025-09-07T09:05:01.5501751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5502143Z     return self._forward_impl(x)
2025-09-07T09:05:01.5502495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5502839Z     x = self.layer3(x)
2025-09-07T09:05:01.5503329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5503699Z     out = self.conv1(x)
2025-09-07T09:05:01.5503794Z 
2025-09-07T09:05:01.5503902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5504258Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5504580Z     return mod(*inputs)
2025-09-07T09:05:01.5504891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5505235Z     return self._forward_impl(x)
2025-09-07T09:05:01.5507321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5507687Z     x = self.layer3(x)
2025-09-07T09:05:01.5507995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5508332Z     out = self.relu(out)
2025-09-07T09:05:01.5508433Z 
2025-09-07T09:05:01.5508608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5508960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5509284Z     return mod(*inputs)
2025-09-07T09:05:01.5509593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5509940Z     return self._forward_impl(x)
2025-09-07T09:05:01.5510286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5510630Z     x = self.layer3(x)
2025-09-07T09:05:01.5510927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5511261Z     out = self.conv2(out)
2025-09-07T09:05:01.5511363Z 
2025-09-07T09:05:01.5511466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5511804Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5512116Z     return mod(*inputs)
2025-09-07T09:05:01.5512422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5512765Z     return self._forward_impl(x)
2025-09-07T09:05:01.5513100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5513440Z     x = self.layer3(x)
2025-09-07T09:05:01.5513740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 100, in forward
2025-09-07T09:05:01.5514081Z     identity = self.downsample(x)
2025-09-07T09:05:01.5514198Z 
2025-09-07T09:05:01.5514299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5514639Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5514951Z     return mod(*inputs)
2025-09-07T09:05:01.5515257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5515590Z     return self._forward_impl(x)
2025-09-07T09:05:01.5515927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5516340Z     x = self.layer3(x)
2025-09-07T09:05:01.5516643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5516974Z     out = self.relu(out)
2025-09-07T09:05:01.5517129Z 
2025-09-07T09:05:01.5517228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5517558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5517868Z     return mod(*inputs)
2025-09-07T09:05:01.5518165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5518501Z     return self._forward_impl(x)
2025-09-07T09:05:01.5518835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5519177Z     x = self.layer3(x)
2025-09-07T09:05:01.5519473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5519804Z     out = self.conv1(x)
2025-09-07T09:05:01.5519893Z 
2025-09-07T09:05:01.5519989Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5520353Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5520668Z     return mod(*inputs)
2025-09-07T09:05:01.5520967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5521311Z     return self._forward_impl(x)
2025-09-07T09:05:01.5521674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5522018Z     x = self.layer3(x)
2025-09-07T09:05:01.5522317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5522645Z     out = self.relu(out)
2025-09-07T09:05:01.5522741Z 
2025-09-07T09:05:01.5522838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5523169Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5523478Z     return mod(*inputs)
2025-09-07T09:05:01.5523777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5524129Z     return self._forward_impl(x)
2025-09-07T09:05:01.5524468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5524809Z     x = self.layer3(x)
2025-09-07T09:05:01.5525107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5525445Z     out = self.conv2(out)
2025-09-07T09:05:01.5525545Z 
2025-09-07T09:05:01.5525652Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5525993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5526302Z     return mod(*inputs)
2025-09-07T09:05:01.5526607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5526948Z     return self._forward_impl(x)
2025-09-07T09:05:01.5527288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 275, in _forward_impl
2025-09-07T09:05:01.5527626Z     x = self.layer3(x)
2025-09-07T09:05:01.5527928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5528259Z     out = self.relu(out)
2025-09-07T09:05:01.5528357Z 
2025-09-07T09:05:01.5528458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5528791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5529147Z     return mod(*inputs)
2025-09-07T09:05:01.5529447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5529788Z     return self._forward_impl(x)
2025-09-07T09:05:01.5530123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5530502Z     x = self.layer4(x)
2025-09-07T09:05:01.5530800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5531128Z     out = self.conv1(x)
2025-09-07T09:05:01.5531217Z 
2025-09-07T09:05:01.5531314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5531649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5531961Z     return mod(*inputs)
2025-09-07T09:05:01.5532260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5532600Z     return self._forward_impl(x)
2025-09-07T09:05:01.5532936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5533272Z     x = self.layer4(x)
2025-09-07T09:05:01.5533604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5533939Z     out = self.relu(out)
2025-09-07T09:05:01.5534032Z 
2025-09-07T09:05:01.5534130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5534463Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5534769Z     return mod(*inputs)
2025-09-07T09:05:01.5535103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5535446Z     return self._forward_impl(x)
2025-09-07T09:05:01.5535782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5536119Z     x = self.layer4(x)
2025-09-07T09:05:01.5536422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5536753Z     out = self.conv2(out)
2025-09-07T09:05:01.5536855Z 
2025-09-07T09:05:01.5536952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5537279Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5537587Z     return mod(*inputs)
2025-09-07T09:05:01.5537949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5538287Z     return self._forward_impl(x)
2025-09-07T09:05:01.5538618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5538956Z     x = self.layer4(x)
2025-09-07T09:05:01.5539252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 100, in forward
2025-09-07T09:05:01.5539596Z     identity = self.downsample(x)
2025-09-07T09:05:01.5539712Z 
2025-09-07T09:05:01.5539812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5540148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5540458Z     return mod(*inputs)
2025-09-07T09:05:01.5540755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5541098Z     return self._forward_impl(x)
2025-09-07T09:05:01.5541436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5541771Z     x = self.layer4(x)
2025-09-07T09:05:01.5542070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 103, in forward
2025-09-07T09:05:01.5542453Z     out = self.relu(out)
2025-09-07T09:05:01.5542551Z 
2025-09-07T09:05:01.5542653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5542985Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5543354Z     return mod(*inputs)
2025-09-07T09:05:01.5543675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5544020Z     return self._forward_impl(x)
2025-09-07T09:05:01.5544361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5544701Z     x = self.layer4(x)
2025-09-07T09:05:01.5544999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 92, in forward
2025-09-07T09:05:01.5545325Z     out = self.conv1(x)
2025-09-07T09:05:01.5545415Z 
2025-09-07T09:05:01.5545517Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5545856Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5546163Z     return mod(*inputs)
2025-09-07T09:05:01.5546465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5546838Z     return self._forward_impl(x)
2025-09-07T09:05:01.5547175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5547512Z     x = self.layer4(x)
2025-09-07T09:05:01.5547812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 94, in forward
2025-09-07T09:05:01.5548176Z     out = self.relu(out)
2025-09-07T09:05:01.5548272Z 
2025-09-07T09:05:01.5548369Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5548701Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5549016Z     return mod(*inputs)
2025-09-07T09:05:01.5549317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5549654Z     return self._forward_impl(x)
2025-09-07T09:05:01.5549992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 276, in _forward_impl
2025-09-07T09:05:01.5550329Z     x = self.layer4(x)
2025-09-07T09:05:01.5550627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 96, in forward
2025-09-07T09:05:01.5550956Z     out = self.conv2(out)
2025-09-07T09:05:01.5551057Z 
2025-09-07T09:05:01.5551156Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5551482Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5551791Z     return mod(*inputs)
2025-09-07T09:05:01.5552101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5552434Z     return self._forward_impl(x)
2025-09-07T09:05:01.5552765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 278, in _forward_impl
2025-09-07T09:05:01.5553114Z     x = self.avgpool(x)
2025-09-07T09:05:01.5553215Z 
2025-09-07T09:05:01.5553306Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:01.5553651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:01.5553965Z     return mod(*inputs)
2025-09-07T09:05:01.5554269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-09-07T09:05:01.5554613Z     return self._forward_impl(x)
2025-09-07T09:05:01.5554946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torchvision/models/resnet.py", line 280, in _forward_impl
2025-09-07T09:05:01.5555337Z     x = self.fc(x)
2025-09-07T09:05:01.5555424Z 
2025-09-07T09:05:18.3255542Z 
2025-09-07T09:05:18.4419424Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:05:18.5472198Z running benchmark:  20% 6/30 [00:00<00:00, 51.68it/s]
2025-09-07T09:05:18.6522377Z running benchmark:  40% 12/30 [00:00<00:00, 54.66it/s]
2025-09-07T09:05:18.7566371Z running benchmark:  60% 18/30 [00:00<00:00, 55.77it/s]
2025-09-07T09:05:18.8610221Z running benchmark:  80% 24/30 [00:00<00:00, 56.42it/s]
2025-09-07T09:05:18.8610570Z running benchmark: 100% 30/30 [00:00<00:00, 56.81it/s]
2025-09-07T09:05:18.8610847Z running benchmark: 100% 30/30 [00:00<00:00, 56.03it/s]
2025-09-07T09:05:18.9159004Z 0.895x
2025-09-07T09:05:18.9163245Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:05:20.9575271Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:05:20.9576255Z   import pynvml  # type: ignore[import]
2025-09-07T09:05:22.9812324Z 
2025-09-07T09:05:23.1627657Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:05:23.1627938Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:05:23.1631866Z cpu  eval  functorch_maml_omniglot            
2025-09-07T09:05:25.5855325Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5855785Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5856132Z     return mod(*inputs)
2025-09-07T09:05:25.5856242Z 
2025-09-07T09:05:25.5856325Z cudagraph partition due to non gpu ops
2025-09-07T09:05:25.5856893Z cudagraph partition due to non gpu ops
2025-09-07T09:05:25.5857135Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5857560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5857905Z     return mod(*inputs)
2025-09-07T09:05:25.5858027Z 
2025-09-07T09:05:25.5858126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5858485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5858820Z     return mod(*inputs)
2025-09-07T09:05:25.5858920Z 
2025-09-07T09:05:25.5859023Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5859361Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5859677Z     return mod(*inputs)
2025-09-07T09:05:25.5859773Z 
2025-09-07T09:05:25.5859867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5860210Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5860522Z     return mod(*inputs)
2025-09-07T09:05:25.5860615Z 
2025-09-07T09:05:25.5860704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5861037Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5861347Z     return mod(*inputs)
2025-09-07T09:05:25.5861436Z 
2025-09-07T09:05:25.5861531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5861872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5862172Z     return mod(*inputs)
2025-09-07T09:05:25.5862269Z 
2025-09-07T09:05:25.5862359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5862694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5863002Z     return mod(*inputs)
2025-09-07T09:05:25.5863092Z 
2025-09-07T09:05:25.5863181Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5863514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5863926Z     return mod(*inputs)
2025-09-07T09:05:25.5864022Z 
2025-09-07T09:05:25.5864126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:25.5864474Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:05:25.5864877Z     return mod(*inputs)
2025-09-07T09:05:25.5864982Z 
2025-09-07T09:05:32.3410347Z 
2025-09-07T09:05:32.4207205Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:05:32.4207529Z running benchmark: 100% 30/30 [00:00<00:00, 379.09it/s]
2025-09-07T09:05:32.4664147Z 0.943x
2025-09-07T09:05:32.4669344Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:05:34.5110361Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:05:34.5111328Z   import pynvml  # type: ignore[import]
2025-09-07T09:05:36.5260889Z 
2025-09-07T09:05:38.0627222Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:05:38.0627518Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:05:38.0655291Z cpu  eval  hf_Albert                          
2025-09-07T09:05:43.3274078Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3274416Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3274645Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3274851Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3275042Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3275241Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3275540Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3275749Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3275943Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3276143Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3276334Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3276542Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3276773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3277159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3277518Z     return mod(**inputs)
2025-09-07T09:05:43.3277927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3278354Z     outputs = self.albert(
2025-09-07T09:05:43.3278752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3279155Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3279592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 520, in forward
2025-09-07T09:05:43.3280038Z     hidden_states = self.embedding_hidden_mapping_in(hidden_states)
2025-09-07T09:05:43.3280223Z 
2025-09-07T09:05:43.3280336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3280704Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3281026Z     return mod(**inputs)
2025-09-07T09:05:43.3281583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3281985Z     outputs = self.albert(
2025-09-07T09:05:43.3282357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3282751Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3283133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3283657Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3284088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3284699Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3285325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3285833Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3286346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3286735Z     self.query(hidden_states)
2025-09-07T09:05:43.3286849Z 
2025-09-07T09:05:43.3286961Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3287314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3287652Z     return mod(**inputs)
2025-09-07T09:05:43.3288041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3288441Z     outputs = self.albert(
2025-09-07T09:05:43.3288846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3289236Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3289624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3290055Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3290516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3291035Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3291553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3292066Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3292575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3292967Z     self.key(hidden_states)
2025-09-07T09:05:43.3293079Z 
2025-09-07T09:05:43.3293185Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3293543Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3293863Z     return mod(**inputs)
2025-09-07T09:05:43.3294221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3294608Z     outputs = self.albert(
2025-09-07T09:05:43.3294969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3295353Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3295736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3296161Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3296587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3297095Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3297691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3298200Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3298753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3299143Z     self.value(hidden_states)
2025-09-07T09:05:43.3299252Z 
2025-09-07T09:05:43.3299330Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3299609Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3299962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3300281Z     return mod(**inputs)
2025-09-07T09:05:43.3300643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3301021Z     outputs = self.albert(
2025-09-07T09:05:43.3301389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3301962Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3302344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3302771Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3303380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3303906Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3304418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3305000Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3305515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3305980Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3306184Z 
2025-09-07T09:05:43.3306287Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3306635Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3306954Z     return mod(**inputs)
2025-09-07T09:05:43.3307313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3307689Z     outputs = self.albert(
2025-09-07T09:05:43.3308050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3308462Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3308854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3309284Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3309703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3310211Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3310721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3311229Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3311729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3312146Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3312312Z 
2025-09-07T09:05:43.3312410Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3312819Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3313131Z     return mod(**inputs)
2025-09-07T09:05:43.3313479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3313863Z     outputs = self.albert(
2025-09-07T09:05:43.3314272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3314660Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3315037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3315451Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3315866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3316377Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3316887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3317282Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3317712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3318119Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3318509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3318910Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3319037Z 
2025-09-07T09:05:43.3319172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3319518Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3319834Z     return mod(**inputs)
2025-09-07T09:05:43.3320188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3320574Z     outputs = self.albert(
2025-09-07T09:05:43.3320927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3321317Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3321690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3322109Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3322533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3323038Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3323546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3323941Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3324328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3324722Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3325100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3325500Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3325854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3326313Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3326552Z 
2025-09-07T09:05:43.3326726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3327074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3327390Z     return mod(**inputs)
2025-09-07T09:05:43.3327756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3328191Z     outputs = self.albert(
2025-09-07T09:05:43.3328547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3328935Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3329313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3329738Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3330156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3330663Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3331170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3331603Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3331996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3332389Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3332771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3333202Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3333345Z 
2025-09-07T09:05:43.3333440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3333789Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3334101Z     return mod(**inputs)
2025-09-07T09:05:43.3334446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3334830Z     outputs = self.albert(
2025-09-07T09:05:43.3335194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3335580Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3335952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3336388Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3336809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3337323Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3337885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3338387Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3338894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3339281Z     self.query(hidden_states)
2025-09-07T09:05:43.3339387Z 
2025-09-07T09:05:43.3339491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3339843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3340150Z     return mod(**inputs)
2025-09-07T09:05:43.3340505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3340941Z     outputs = self.albert(
2025-09-07T09:05:43.3341302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3341690Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3342060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3342526Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3342947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3343459Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3343972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3344468Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3344968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3354262Z     self.key(hidden_states)
2025-09-07T09:05:43.3354416Z 
2025-09-07T09:05:43.3354629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3355024Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3355368Z     return mod(**inputs)
2025-09-07T09:05:43.3355767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3356180Z     outputs = self.albert(
2025-09-07T09:05:43.3356594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3356999Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3357389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3357828Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3358260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3358783Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3359301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3359813Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3360320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3360702Z     self.value(hidden_states)
2025-09-07T09:05:43.3360823Z 
2025-09-07T09:05:43.3360905Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3361148Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3361519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3361849Z     return mod(**inputs)
2025-09-07T09:05:43.3362234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3362636Z     outputs = self.albert(
2025-09-07T09:05:43.3362998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3363394Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3363770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3364246Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3364672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3365194Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3365764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3366291Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3366800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3367266Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3367469Z 
2025-09-07T09:05:43.3367572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3367930Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3368242Z     return mod(**inputs)
2025-09-07T09:05:43.3368606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3368994Z     outputs = self.albert(
2025-09-07T09:05:43.3369388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3369783Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3370151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3370617Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3371047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3371560Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3372068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3372565Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3373066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3373489Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3373654Z 
2025-09-07T09:05:43.3373759Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3374109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3374424Z     return mod(**inputs)
2025-09-07T09:05:43.3374777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3375162Z     outputs = self.albert(
2025-09-07T09:05:43.3375518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3375897Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3376274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3376697Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3377122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3377717Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3378224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3378677Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3379079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3379479Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3379873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3380371Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3380507Z 
2025-09-07T09:05:43.3380606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3380956Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3381276Z     return mod(**inputs)
2025-09-07T09:05:43.3381637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3382014Z     outputs = self.albert(
2025-09-07T09:05:43.3382375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3382761Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3383137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3384582Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3385010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3385518Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3386064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3386472Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3386864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3387258Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3387648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3388057Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3388406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3388856Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3389099Z 
2025-09-07T09:05:43.3389200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3389555Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3389872Z     return mod(**inputs)
2025-09-07T09:05:43.3390237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3390625Z     outputs = self.albert(
2025-09-07T09:05:43.3390987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3391377Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3391760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3392182Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3392601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3393114Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3393671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3394065Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3394450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3394883Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3395267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3395668Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3395795Z 
2025-09-07T09:05:43.3395900Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3396242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3396560Z     return mod(**inputs)
2025-09-07T09:05:43.3396910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3397298Z     outputs = self.albert(
2025-09-07T09:05:43.3397656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3398036Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3398449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3398874Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3399297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3399839Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3400347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3400853Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3401356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3401743Z     self.query(hidden_states)
2025-09-07T09:05:43.3401853Z 
2025-09-07T09:05:43.3401956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3402294Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3402604Z     return mod(**inputs)
2025-09-07T09:05:43.3402959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3403528Z     outputs = self.albert(
2025-09-07T09:05:43.3403882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3404271Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3404644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3405071Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3405486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3405988Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3406497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3406999Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3407499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3407993Z     self.key(hidden_states)
2025-09-07T09:05:43.3408100Z 
2025-09-07T09:05:43.3408200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3408550Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3408864Z     return mod(**inputs)
2025-09-07T09:05:43.3409275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3409659Z     outputs = self.albert(
2025-09-07T09:05:43.3410010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3410400Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3410779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3411203Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3411619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3412126Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3412682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3413203Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3413713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3414094Z     self.value(hidden_states)
2025-09-07T09:05:43.3414266Z 
2025-09-07T09:05:43.3414343Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3414575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3414933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3415250Z     return mod(**inputs)
2025-09-07T09:05:43.3415606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3415991Z     outputs = self.albert(
2025-09-07T09:05:43.3416351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3416742Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3417112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3417580Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3418007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3418521Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3419029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3419538Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3420034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3420496Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3420695Z 
2025-09-07T09:05:43.3420789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3421133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3421451Z     return mod(**inputs)
2025-09-07T09:05:43.3421798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3422224Z     outputs = self.albert(
2025-09-07T09:05:43.3422573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3422957Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3423363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3423780Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3424189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3424692Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3425190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3425682Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3426174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3426586Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3426785Z 
2025-09-07T09:05:43.3426880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3427221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3427536Z     return mod(**inputs)
2025-09-07T09:05:43.3427907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3428295Z     outputs = self.albert(
2025-09-07T09:05:43.3428640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3429018Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3429374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3429785Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3430197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3430696Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3431193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3431583Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3431974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3432361Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3432741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3433130Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3433257Z 
2025-09-07T09:05:43.3433355Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3433695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3433995Z     return mod(**inputs)
2025-09-07T09:05:43.3434343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3434733Z     outputs = self.albert(
2025-09-07T09:05:43.3435137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3435529Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3435939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3436352Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3436757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3437293Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3437795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3438199Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3438603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3439007Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3439420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3439816Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3440172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3440671Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3440905Z 
2025-09-07T09:05:43.3441002Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3441360Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3441676Z     return mod(**inputs)
2025-09-07T09:05:43.3442074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3442462Z     outputs = self.albert(
2025-09-07T09:05:43.3442819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3443203Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3443574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3443993Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3444409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3444915Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3445419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3445826Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3446225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3446611Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3447004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3447402Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3447535Z 
2025-09-07T09:05:43.3447648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3447991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3448296Z     return mod(**inputs)
2025-09-07T09:05:43.3448669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3449056Z     outputs = self.albert(
2025-09-07T09:05:43.3449422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3449863Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3450263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3450710Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3451166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3451687Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3452203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3452704Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3453198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3453577Z     self.query(hidden_states)
2025-09-07T09:05:43.3453684Z 
2025-09-07T09:05:43.3453783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3454127Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3454498Z     return mod(**inputs)
2025-09-07T09:05:43.3454849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3455246Z     outputs = self.albert(
2025-09-07T09:05:43.3455608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3456049Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3456423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3456847Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3457283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3457824Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3458333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3458829Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3459346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3459748Z     self.key(hidden_states)
2025-09-07T09:05:43.3459852Z 
2025-09-07T09:05:43.3459950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3460306Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3460636Z     return mod(**inputs)
2025-09-07T09:05:43.3460979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3461352Z     outputs = self.albert(
2025-09-07T09:05:43.3461706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3462083Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3462451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3462860Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3463295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3463872Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3464404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3464899Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3465438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3465819Z     self.value(hidden_states)
2025-09-07T09:05:43.3465927Z 
2025-09-07T09:05:43.3465999Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3466220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3466562Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3466868Z     return mod(**inputs)
2025-09-07T09:05:43.3467212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3467590Z     outputs = self.albert(
2025-09-07T09:05:43.3467942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3468319Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3468726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3469148Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3469564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3470097Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3470602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3471100Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3471589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3472042Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3472239Z 
2025-09-07T09:05:43.3472332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3472673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3472978Z     return mod(**inputs)
2025-09-07T09:05:43.3473322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3473693Z     outputs = self.albert(
2025-09-07T09:05:43.3474040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3474420Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3474785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3475195Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3475613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3476115Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3476611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3477099Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3477583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3478034Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3478194Z 
2025-09-07T09:05:43.3478286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3478621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3478964Z     return mod(**inputs)
2025-09-07T09:05:43.3479306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3479683Z     outputs = self.albert(
2025-09-07T09:05:43.3480034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3480412Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3480774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3481186Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3481597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3482128Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3482633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3483019Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3483410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3483833Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3484222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3484620Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3484744Z 
2025-09-07T09:05:43.3484835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3485169Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3485481Z     return mod(**inputs)
2025-09-07T09:05:43.3485828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3486204Z     outputs = self.albert(
2025-09-07T09:05:43.3486554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3486933Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3487302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3487720Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3488129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3488632Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3489132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3489525Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3489909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3490288Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3490665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3491065Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3491456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3491902Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3492132Z 
2025-09-07T09:05:43.3492227Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3492603Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3492924Z     return mod(**inputs)
2025-09-07T09:05:43.3493284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3493667Z     outputs = self.albert(
2025-09-07T09:05:43.3494017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3494401Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3494776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3495200Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3495613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3496159Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3496669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3497062Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3497545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3497939Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3498326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3498730Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3498861Z 
2025-09-07T09:05:43.3498962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3499312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3499620Z     return mod(**inputs)
2025-09-07T09:05:43.3499972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3500354Z     outputs = self.albert(
2025-09-07T09:05:43.3500709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3501092Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3501455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3501874Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3502290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3502797Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3503483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3503990Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3504507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3504897Z     self.query(hidden_states)
2025-09-07T09:05:43.3505009Z 
2025-09-07T09:05:43.3505116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3505552Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3505868Z     return mod(**inputs)
2025-09-07T09:05:43.3506230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3506671Z     outputs = self.albert(
2025-09-07T09:05:43.3507046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3507434Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3507818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3508250Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3508675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3509188Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3509691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3510257Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3510777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3511164Z     self.key(hidden_states)
2025-09-07T09:05:43.3511271Z 
2025-09-07T09:05:43.3511376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3511772Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3512094Z     return mod(**inputs)
2025-09-07T09:05:43.3512457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3512846Z     outputs = self.albert(
2025-09-07T09:05:43.3513205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3513595Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3513977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3514404Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3514825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3515337Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3515852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3516355Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3516858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3517242Z     self.value(hidden_states)
2025-09-07T09:05:43.3517351Z 
2025-09-07T09:05:43.3517428Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3517658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3518009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3518324Z     return mod(**inputs)
2025-09-07T09:05:43.3518683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3519054Z     outputs = self.albert(
2025-09-07T09:05:43.3519411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3519851Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3520226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3520640Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3521100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3521610Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3522118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3522617Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3523109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3523566Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3523766Z 
2025-09-07T09:05:43.3523858Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3524232Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3524553Z     return mod(**inputs)
2025-09-07T09:05:43.3524912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3525305Z     outputs = self.albert(
2025-09-07T09:05:43.3525707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3526107Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3526493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3526916Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3527340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3527858Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3528371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3528877Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3529373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3529789Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3529957Z 
2025-09-07T09:05:43.3530059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3530415Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3530730Z     return mod(**inputs)
2025-09-07T09:05:43.3531088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3531473Z     outputs = self.albert(
2025-09-07T09:05:43.3531833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3532222Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3532594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3533017Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3533434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3533986Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3534499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3534930Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3535336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3535729Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3536123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3536522Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3536650Z 
2025-09-07T09:05:43.3536747Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3537095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3537410Z     return mod(**inputs)
2025-09-07T09:05:43.3537829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3538214Z     outputs = self.albert(
2025-09-07T09:05:43.3538606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3539002Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3539379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3539802Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3540247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3540762Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3541274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3541668Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3542062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3542450Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3542843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3543252Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3543609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3544063Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3544298Z 
2025-09-07T09:05:43.3544394Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3544758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3545073Z     return mod(**inputs)
2025-09-07T09:05:43.3545430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3545816Z     outputs = self.albert(
2025-09-07T09:05:43.3546168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3546550Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3546937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3547372Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3547830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3548343Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3548858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3549293Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3549694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3550081Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3550478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3550880Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3551016Z 
2025-09-07T09:05:43.3551125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3551481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3551793Z     return mod(**inputs)
2025-09-07T09:05:43.3552159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3552583Z     outputs = self.albert(
2025-09-07T09:05:43.3552948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3553343Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3553715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3554172Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3554593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3555114Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3555625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3556130Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3556632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3557019Z     self.query(hidden_states)
2025-09-07T09:05:43.3557127Z 
2025-09-07T09:05:43.3557232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3557577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3557890Z     return mod(**inputs)
2025-09-07T09:05:43.3558244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3558629Z     outputs = self.albert(
2025-09-07T09:05:43.3558984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3559365Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3559743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3560160Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3560577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3561085Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3561584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3562126Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3562627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3563276Z     self.key(hidden_states)
2025-09-07T09:05:43.3563389Z 
2025-09-07T09:05:43.3563494Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3563832Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3564145Z     return mod(**inputs)
2025-09-07T09:05:43.3564506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3564896Z     outputs = self.albert(
2025-09-07T09:05:43.3565245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3565635Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3566009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3566425Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3566890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3567397Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3567906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3568438Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3568965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3569356Z     self.value(hidden_states)
2025-09-07T09:05:43.3569460Z 
2025-09-07T09:05:43.3569538Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3569766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3570112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3570430Z     return mod(**inputs)
2025-09-07T09:05:43.3570782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3571160Z     outputs = self.albert(
2025-09-07T09:05:43.3571518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3571901Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3572272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3572686Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3573100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3573612Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3574119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3574614Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3575114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3575567Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3575767Z 
2025-09-07T09:05:43.3575898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3576238Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3576554Z     return mod(**inputs)
2025-09-07T09:05:43.3576900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3577335Z     outputs = self.albert(
2025-09-07T09:05:43.3577756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3578153Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3578528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3578943Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3579364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3579886Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3580399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3580943Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3581448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3581878Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3582047Z 
2025-09-07T09:05:43.3582176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3582526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3582843Z     return mod(**inputs)
2025-09-07T09:05:43.3583195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3583577Z     outputs = self.albert(
2025-09-07T09:05:43.3583933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3584318Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3584685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3585111Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3585530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3586034Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3586540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3586933Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3587327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3587727Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3588122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3588516Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3588642Z 
2025-09-07T09:05:43.3588737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3589083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3589396Z     return mod(**inputs)
2025-09-07T09:05:43.3589753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3590181Z     outputs = self.albert(
2025-09-07T09:05:43.3590531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3590916Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3591326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3591749Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3592160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3592667Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3593172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3593563Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3593954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3594337Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3594752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3595163Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3595514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3595966Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3596234Z 
2025-09-07T09:05:43.3596333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3596680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3596998Z     return mod(**inputs)
2025-09-07T09:05:43.3597355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3597737Z     outputs = self.albert(
2025-09-07T09:05:43.3598088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3598478Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3598853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3599275Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3599698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3600198Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3600709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3601101Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3601489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3601876Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3602262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3602655Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3602784Z 
2025-09-07T09:05:43.3602886Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3603366Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3603674Z     return mod(**inputs)
2025-09-07T09:05:43.3604161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3604540Z     outputs = self.albert(
2025-09-07T09:05:43.3604889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3605355Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3605739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3606168Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3606588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3607092Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3607601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3608102Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3608613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3609042Z     self.query(hidden_states)
2025-09-07T09:05:43.3609154Z 
2025-09-07T09:05:43.3609260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3609612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3609922Z     return mod(**inputs)
2025-09-07T09:05:43.3610331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3610725Z     outputs = self.albert(
2025-09-07T09:05:43.3611089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3611470Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3611847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3612273Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3612695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3613204Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3613706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3614215Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3614717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3615100Z     self.key(hidden_states)
2025-09-07T09:05:43.3615206Z 
2025-09-07T09:05:43.3615312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3615654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3615968Z     return mod(**inputs)
2025-09-07T09:05:43.3616325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3616710Z     outputs = self.albert(
2025-09-07T09:05:43.3617059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3617445Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3617865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3618348Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3618768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3619273Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3619826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3620323Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3620825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3621214Z     self.value(hidden_states)
2025-09-07T09:05:43.3621322Z 
2025-09-07T09:05:43.3621397Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3621630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3621979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3622291Z     return mod(**inputs)
2025-09-07T09:05:43.3622639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3623053Z     outputs = self.albert(
2025-09-07T09:05:43.3623414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3623796Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3624204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3624622Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3625044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3625553Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3626056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3626670Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3627298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3627780Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3628001Z 
2025-09-07T09:05:43.3628099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3628457Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3628775Z     return mod(**inputs)
2025-09-07T09:05:43.3629141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3629519Z     outputs = self.albert(
2025-09-07T09:05:43.3629874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3630263Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3630634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3631046Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3631464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3631990Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3632566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3633079Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3633615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3634108Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3634275Z 
2025-09-07T09:05:43.3634370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3634760Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3635154Z     return mod(**inputs)
2025-09-07T09:05:43.3635553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3635952Z     outputs = self.albert(
2025-09-07T09:05:43.3636306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3636692Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3637073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3637545Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3637968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3638485Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3639022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3639420Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3639813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3640208Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3640589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3640990Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3641124Z 
2025-09-07T09:05:43.3641221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3641577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3649867Z     return mod(**inputs)
2025-09-07T09:05:43.3650293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3650714Z     outputs = self.albert(
2025-09-07T09:05:43.3651108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3651523Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3651913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3652343Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3652772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3653282Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3653789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3654183Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3654594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3655063Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3655454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3655856Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3656201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3656695Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3656935Z 
2025-09-07T09:05:43.3657040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3657394Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3657790Z     return mod(**inputs)
2025-09-07T09:05:43.3658159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3658553Z     outputs = self.albert(
2025-09-07T09:05:43.3658910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3659291Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3659702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3660136Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3660559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3661073Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3661612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3662015Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3662429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3662828Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3663213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3663618Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3663754Z 
2025-09-07T09:05:43.3663854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3664207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3664519Z     return mod(**inputs)
2025-09-07T09:05:43.3664871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3665255Z     outputs = self.albert(
2025-09-07T09:05:43.3665611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3666006Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3666382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3666801Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3667221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3667724Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3668228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3668729Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3669278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3669661Z     self.query(hidden_states)
2025-09-07T09:05:43.3669777Z 
2025-09-07T09:05:43.3669876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3670223Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3670568Z     return mod(**inputs)
2025-09-07T09:05:43.3670928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3671302Z     outputs = self.albert(
2025-09-07T09:05:43.3671655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3672033Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3672397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3672814Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3673223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3673756Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3674262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3674751Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3675293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3675672Z     self.key(hidden_states)
2025-09-07T09:05:43.3675777Z 
2025-09-07T09:05:43.3675875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3676214Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3676514Z     return mod(**inputs)
2025-09-07T09:05:43.3676861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3677234Z     outputs = self.albert(
2025-09-07T09:05:43.3677586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3677959Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3678322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3678734Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3679142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3679649Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3680150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3680648Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3681139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3681516Z     self.value(hidden_states)
2025-09-07T09:05:43.3681623Z 
2025-09-07T09:05:43.3681705Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3681925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3682265Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3682611Z     return mod(**inputs)
2025-09-07T09:05:43.3682963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3683343Z     outputs = self.albert(
2025-09-07T09:05:43.3683697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3684114Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3684482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3684900Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3685307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3685811Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3686314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3686804Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3687300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3687791Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3687987Z 
2025-09-07T09:05:43.3688081Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3688419Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3688725Z     return mod(**inputs)
2025-09-07T09:05:43.3689107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3689480Z     outputs = self.albert(
2025-09-07T09:05:43.3689837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3690217Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3690585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3691000Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3691407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3691908Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3692414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3692904Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3693396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3693805Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3693970Z 
2025-09-07T09:05:43.3694061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3694401Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3694704Z     return mod(**inputs)
2025-09-07T09:05:43.3695049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3695420Z     outputs = self.albert(
2025-09-07T09:05:43.3695769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3696149Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3696561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3696974Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3697385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3697991Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3698498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3698892Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3699293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3699690Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3700079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3700495Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3700624Z 
2025-09-07T09:05:43.3700730Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3701076Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3701431Z     return mod(**inputs)
2025-09-07T09:05:43.3701792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3702179Z     outputs = self.albert(
2025-09-07T09:05:43.3702541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3702957Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3703495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3703922Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3704345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3704847Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3705396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3705788Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3706177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3706570Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3706956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3707356Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3707697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3708145Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3708379Z 
2025-09-07T09:05:43.3708482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3708835Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3709143Z     return mod(**inputs)
2025-09-07T09:05:43.3709490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3709872Z     outputs = self.albert(
2025-09-07T09:05:43.3710230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3710688Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3711059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3711480Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3711913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3712500Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3713009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3713407Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3713802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3714188Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3714571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3714965Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3715105Z 
2025-09-07T09:05:43.3715203Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3715607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3715928Z     return mod(**inputs)
2025-09-07T09:05:43.3716285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3716657Z     outputs = self.albert(
2025-09-07T09:05:43.3717060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3717451Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3717835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3718266Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3718685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3719200Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3719706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3720209Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3720719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3721106Z     self.query(hidden_states)
2025-09-07T09:05:43.3721218Z 
2025-09-07T09:05:43.3721316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3721662Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3721968Z     return mod(**inputs)
2025-09-07T09:05:43.3722320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3722698Z     outputs = self.albert(
2025-09-07T09:05:43.3723054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3723436Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3723803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3724214Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3724622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3725175Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3725681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3726227Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3726720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3727094Z     self.key(hidden_states)
2025-09-07T09:05:43.3727204Z 
2025-09-07T09:05:43.3727296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3727636Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3727946Z     return mod(**inputs)
2025-09-07T09:05:43.3728288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3728660Z     outputs = self.albert(
2025-09-07T09:05:43.3729007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3729437Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3729803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3730217Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3730626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3731159Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3731663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3732186Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3732686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3733064Z     self.value(hidden_states)
2025-09-07T09:05:43.3733166Z 
2025-09-07T09:05:43.3733243Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3733462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3733791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3734096Z     return mod(**inputs)
2025-09-07T09:05:43.3734443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3734814Z     outputs = self.albert(
2025-09-07T09:05:43.3735169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3735543Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3735907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3736320Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3736725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3737220Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3737777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3738271Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3738809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3739262Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3739454Z 
2025-09-07T09:05:43.3739551Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3739920Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3740228Z     return mod(**inputs)
2025-09-07T09:05:43.3740577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3740951Z     outputs = self.albert(
2025-09-07T09:05:43.3741299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3741676Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3742040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3742451Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3742853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3743379Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3743880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3744369Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3744899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3745325Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3745486Z 
2025-09-07T09:05:43.3745577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3745914Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3746222Z     return mod(**inputs)
2025-09-07T09:05:43.3746565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3746934Z     outputs = self.albert(
2025-09-07T09:05:43.3747276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3747651Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3748018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3748428Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3748838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3749338Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3749836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3750234Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3750619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3751002Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3751383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3751771Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3751897Z 
2025-09-07T09:05:43.3752000Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3752693Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3753008Z     return mod(**inputs)
2025-09-07T09:05:43.3753350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3753765Z     outputs = self.albert(
2025-09-07T09:05:43.3754118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3754490Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3754853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3755270Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3755676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3756175Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3756682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3757076Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3757504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3757894Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3758274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3758716Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3759056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3759500Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3759730Z 
2025-09-07T09:05:43.3759834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3760182Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3760491Z     return mod(**inputs)
2025-09-07T09:05:43.3760846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3761226Z     outputs = self.albert(
2025-09-07T09:05:43.3761585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3761969Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3762335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3762752Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3763165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3763667Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3764170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3764554Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3764940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3765326Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3765700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3766086Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3766261Z 
2025-09-07T09:05:43.3766354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3766692Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3766995Z     return mod(**inputs)
2025-09-07T09:05:43.3767340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3767756Z     outputs = self.albert(
2025-09-07T09:05:43.3768124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3768513Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3768889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3769302Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3769710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3770214Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3770751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3771255Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3771743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3772116Z     self.query(hidden_states)
2025-09-07T09:05:43.3772223Z 
2025-09-07T09:05:43.3772347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3772695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3773000Z     return mod(**inputs)
2025-09-07T09:05:43.3773353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3773753Z     outputs = self.albert(
2025-09-07T09:05:43.3774112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3774494Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3774880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3775299Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3775706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3776216Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3776734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3777224Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3777801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3778185Z     self.key(hidden_states)
2025-09-07T09:05:43.3778293Z 
2025-09-07T09:05:43.3778383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3778719Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3779021Z     return mod(**inputs)
2025-09-07T09:05:43.3779370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3779756Z     outputs = self.albert(
2025-09-07T09:05:43.3780160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3780544Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3780921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3781397Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3781832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3782361Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3782872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3783375Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3783870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3784263Z     self.value(hidden_states)
2025-09-07T09:05:43.3784375Z 
2025-09-07T09:05:43.3784454Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3784693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3785069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3785386Z     return mod(**inputs)
2025-09-07T09:05:43.3785745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3786122Z     outputs = self.albert(
2025-09-07T09:05:43.3786509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3786890Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3787259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3787674Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3788085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3788590Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3789086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3789577Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3790071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3790523Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3790715Z 
2025-09-07T09:05:43.3790812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3791142Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3791452Z     return mod(**inputs)
2025-09-07T09:05:43.3791794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3792174Z     outputs = self.albert(
2025-09-07T09:05:43.3792519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3792897Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3793271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3793682Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3794138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3794631Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3795135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3795663Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3796153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3796564Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3796725Z 
2025-09-07T09:05:43.3796815Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3797150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3797456Z     return mod(**inputs)
2025-09-07T09:05:43.3797800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3798176Z     outputs = self.albert(
2025-09-07T09:05:43.3798552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3798934Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3799297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3799713Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3800151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3800655Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3801155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3801541Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3801930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3802315Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3802697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3803248Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3803378Z 
2025-09-07T09:05:43.3803475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3803812Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3804115Z     return mod(**inputs)
2025-09-07T09:05:43.3804464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3804839Z     outputs = self.albert(
2025-09-07T09:05:43.3805184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3805569Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3805933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3806356Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3806787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3807308Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3807814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3808276Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3808673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3809061Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3809506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3809906Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3810247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3810697Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3810928Z 
2025-09-07T09:05:43.3811093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3811447Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3811758Z     return mod(**inputs)
2025-09-07T09:05:43.3812118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3812502Z     outputs = self.albert(
2025-09-07T09:05:43.3812910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3813300Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3813672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3814145Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3814562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3815073Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3815575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3815962Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3816350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3816740Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3817119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3817568Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3817701Z 
2025-09-07T09:05:43.3817796Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3818145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3818455Z     return mod(**inputs)
2025-09-07T09:05:43.3818808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3819185Z     outputs = self.albert(
2025-09-07T09:05:43.3819544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3819933Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3820306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3820730Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3821146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3821649Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3822208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3822710Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3823215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3823635Z     self.query(hidden_states)
2025-09-07T09:05:43.3823750Z 
2025-09-07T09:05:43.3823847Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3824194Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3824508Z     return mod(**inputs)
2025-09-07T09:05:43.3824867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3825243Z     outputs = self.albert(
2025-09-07T09:05:43.3825600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3825982Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3826359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3826807Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3827232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3827739Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3828378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3828884Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3829380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3829763Z     self.key(hidden_states)
2025-09-07T09:05:43.3829875Z 
2025-09-07T09:05:43.3829969Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3830321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3830651Z     return mod(**inputs)
2025-09-07T09:05:43.3830995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3831373Z     outputs = self.albert(
2025-09-07T09:05:43.3831728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3832112Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3832479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3832891Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3833309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3833822Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3834328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3834826Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3835333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3835715Z     self.value(hidden_states)
2025-09-07T09:05:43.3835880Z 
2025-09-07T09:05:43.3835958Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3836189Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3836527Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3836835Z     return mod(**inputs)
2025-09-07T09:05:43.3837203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3837628Z     outputs = self.albert(
2025-09-07T09:05:43.3837987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3838366Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3838736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3839148Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3839563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3840063Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3840597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3841097Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3841591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3842048Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3842273Z 
2025-09-07T09:05:43.3842373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3842714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3843025Z     return mod(**inputs)
2025-09-07T09:05:43.3843377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3843756Z     outputs = self.albert(
2025-09-07T09:05:43.3844105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3844489Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3844862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3845280Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3845696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3846192Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3846700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3847197Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3847694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3848112Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3848272Z 
2025-09-07T09:05:43.3848370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3848703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3849020Z     return mod(**inputs)
2025-09-07T09:05:43.3849367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3849788Z     outputs = self.albert(
2025-09-07T09:05:43.3850135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3850516Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3850888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3851354Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3851761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3852255Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3852753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3853138Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3853526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3853909Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3854287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3854711Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3854842Z 
2025-09-07T09:05:43.3854938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3855279Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3855581Z     return mod(**inputs)
2025-09-07T09:05:43.3855955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3856334Z     outputs = self.albert(
2025-09-07T09:05:43.3856690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3857066Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3857426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3857925Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3858352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3858863Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3859376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3859771Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3860169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3860565Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3860958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3861359Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3861707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3862157Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3862403Z 
2025-09-07T09:05:43.3862505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3862858Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3863188Z     return mod(**inputs)
2025-09-07T09:05:43.3863547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3863983Z     outputs = self.albert(
2025-09-07T09:05:43.3864359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3864747Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3865165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3865590Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3866009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3866524Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3867037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3867428Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3867821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3868213Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3868644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3869049Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3869180Z 
2025-09-07T09:05:43.3869279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3869629Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3869987Z     return mod(**inputs)
2025-09-07T09:05:43.3870345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3870739Z     outputs = self.albert(
2025-09-07T09:05:43.3871100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3871483Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3871860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3872284Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3872697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3873204Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3873713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3874215Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3874717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 380, in forward
2025-09-07T09:05:43.3875095Z     self.query(hidden_states)
2025-09-07T09:05:43.3875209Z 
2025-09-07T09:05:43.3875303Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3875645Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3875959Z     return mod(**inputs)
2025-09-07T09:05:43.3876311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3876685Z     outputs = self.albert(
2025-09-07T09:05:43.3877039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3877418Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3877841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3878254Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3878667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3879215Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3879727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3880238Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3880738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 385, in forward
2025-09-07T09:05:43.3881115Z     self.key(hidden_states)
2025-09-07T09:05:43.3881233Z 
2025-09-07T09:05:43.3881324Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3881667Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3881983Z     return mod(**inputs)
2025-09-07T09:05:43.3882360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3882744Z     outputs = self.albert(
2025-09-07T09:05:43.3883106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3883499Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3883935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3884355Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3884774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3885276Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3885777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3886286Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3886780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 390, in forward
2025-09-07T09:05:43.3887160Z     self.value(hidden_states)
2025-09-07T09:05:43.3887273Z 
2025-09-07T09:05:43.3887352Z cudagraph partition due to non gpu ops
2025-09-07T09:05:43.3887576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3887920Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3888230Z     return mod(**inputs)
2025-09-07T09:05:43.3888581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3888960Z     outputs = self.albert(
2025-09-07T09:05:43.3889317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3889700Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3890068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3890486Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3890899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3891400Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3891953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3892448Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3892946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 403, in forward
2025-09-07T09:05:43.3893451Z     attention_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:05:43.3893645Z 
2025-09-07T09:05:43.3893751Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3894088Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3894403Z     return mod(**inputs)
2025-09-07T09:05:43.3894759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3895143Z     outputs = self.albert(
2025-09-07T09:05:43.3895499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3895879Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3896289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3896715Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3897134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3897681Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3898223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 449, in forward
2025-09-07T09:05:43.3898730Z     attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
2025-09-07T09:05:43.3899227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 415, in forward
2025-09-07T09:05:43.3899647Z     projected_context_layer = self.dense(attention_output)
2025-09-07T09:05:43.3899813Z 
2025-09-07T09:05:43.3899915Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3900252Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3900565Z     return mod(**inputs)
2025-09-07T09:05:43.3900918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3901300Z     outputs = self.albert(
2025-09-07T09:05:43.3901650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3902035Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3902409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3902836Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3903382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3903588Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3903856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3903942Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3904207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3904284Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3904624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 462, in ff_chunk
2025-09-07T09:05:43.3904701Z     ffn_output = self.ffn(attention_output)
2025-09-07T09:05:43.3904709Z 
2025-09-07T09:05:43.3904807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3905057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3905126Z     return mod(**inputs)
2025-09-07T09:05:43.3905387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3905455Z     outputs = self.albert(
2025-09-07T09:05:43.3905711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3905780Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3906038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3906147Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3906407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3906645Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3906911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3906987Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3907304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3907381Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3907642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 463, in ff_chunk
2025-09-07T09:05:43.3907730Z     ffn_output = self.activation(ffn_output)
2025-09-07T09:05:43.3907940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:05:43.3908119Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:05:43.3908124Z 
2025-09-07T09:05:43.3908231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3908424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3908491Z     return mod(**inputs)
2025-09-07T09:05:43.3908753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 971, in forward
2025-09-07T09:05:43.3908822Z     outputs = self.albert(
2025-09-07T09:05:43.3909078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 722, in forward
2025-09-07T09:05:43.3909150Z     encoder_outputs = self.encoder(
2025-09-07T09:05:43.3909409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 534, in forward
2025-09-07T09:05:43.3909519Z     layer_group_output = self.albert_layer_groups[group_idx](
2025-09-07T09:05:43.3909785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 486, in forward
2025-09-07T09:05:43.3909972Z     layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
2025-09-07T09:05:43.3910224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 451, in forward
2025-09-07T09:05:43.3910306Z     ffn_output = apply_chunking_to_forward(
2025-09-07T09:05:43.3910568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:05:43.3910680Z     return forward_fn(*input_tensors)
2025-09-07T09:05:43.3910943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 464, in ff_chunk
2025-09-07T09:05:43.3911026Z     ffn_output = self.ffn_output(ffn_output)
2025-09-07T09:05:43.3911062Z 
2025-09-07T09:05:43.3911158Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3911346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3911411Z     return mod(**inputs)
2025-09-07T09:05:43.3911663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 984, in forward
2025-09-07T09:05:43.3911779Z     prediction_scores = self.predictions(sequence_outputs)
2025-09-07T09:05:43.3912032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 866, in forward
2025-09-07T09:05:43.3912111Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:05:43.3912114Z 
2025-09-07T09:05:43.3912214Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:05:43.3912399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:05:43.3912462Z     return mod(**inputs)
2025-09-07T09:05:43.3912750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 984, in forward
2025-09-07T09:05:43.3912860Z     prediction_scores = self.predictions(sequence_outputs)
2025-09-07T09:05:43.3913112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/albert/modeling_albert.py", line 869, in forward
2025-09-07T09:05:43.3913568Z     hidden_states = self.decoder(hidden_states)
2025-09-07T09:05:43.3913573Z 
2025-09-07T09:05:51.5978035Z 
2025-09-07T09:05:51.7860682Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:05:51.9677647Z running benchmark:   7% 2/30 [00:00<00:02, 10.64it/s]
2025-09-07T09:05:52.1499092Z running benchmark:  13% 4/30 [00:00<00:02, 10.86it/s]
2025-09-07T09:05:52.3316147Z running benchmark:  20% 6/30 [00:00<00:02, 10.91it/s]
2025-09-07T09:05:52.5124016Z running benchmark:  27% 8/30 [00:00<00:02, 10.95it/s]
2025-09-07T09:05:52.6940174Z running benchmark:  33% 10/30 [00:00<00:01, 10.99it/s]
2025-09-07T09:05:52.8750144Z running benchmark:  40% 12/30 [00:01<00:01, 11.00it/s]
2025-09-07T09:05:53.0556115Z running benchmark:  47% 14/30 [00:01<00:01, 11.01it/s]
2025-09-07T09:05:53.2360110Z running benchmark:  53% 16/30 [00:01<00:01, 11.03it/s]
2025-09-07T09:05:53.4164562Z running benchmark:  60% 18/30 [00:01<00:01, 11.05it/s]
2025-09-07T09:05:53.5979695Z running benchmark:  67% 20/30 [00:01<00:00, 11.06it/s]
2025-09-07T09:05:53.7786066Z running benchmark:  73% 22/30 [00:02<00:00, 11.05it/s]
2025-09-07T09:05:53.9609598Z running benchmark:  80% 24/30 [00:02<00:00, 11.06it/s]
2025-09-07T09:05:54.1431047Z running benchmark:  87% 26/30 [00:02<00:00, 11.03it/s]
2025-09-07T09:05:54.3245666Z running benchmark:  93% 28/30 [00:02<00:00, 11.01it/s]
2025-09-07T09:05:54.3246037Z running benchmark: 100% 30/30 [00:02<00:00, 11.02it/s]
2025-09-07T09:05:54.3246320Z running benchmark: 100% 30/30 [00:02<00:00, 11.00it/s]
2025-09-07T09:05:54.3884868Z 1.334x
2025-09-07T09:05:54.3889070Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:05:56.5825462Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:05:56.5833436Z   import pynvml  # type: ignore[import]
2025-09-07T09:05:58.6040735Z 
2025-09-07T09:06:02.4641632Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:06:02.4641980Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:06:02.4885996Z cpu  eval  hf_Bart                            
2025-09-07T09:06:09.2820397Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2820726Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2820945Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2821148Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2821346Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2821889Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2822125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2822518Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2822867Z     return mod(**inputs)
2025-09-07T09:06:09.2823286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2823687Z     outputs = self.model(
2025-09-07T09:06:09.2824054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2824454Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2824840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2825216Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2825660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2826036Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2826492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2826923Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2827417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.2827920Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.2828143Z 
2025-09-07T09:06:09.2828248Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2828621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2828953Z     return mod(**inputs)
2025-09-07T09:06:09.2829316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2829711Z     outputs = self.model(
2025-09-07T09:06:09.2830093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2830489Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2830866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2831253Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2831606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2831975Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2832362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2832755Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2833151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.2833534Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.2833664Z 
2025-09-07T09:06:09.2833772Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2834134Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2834454Z     return mod(**inputs)
2025-09-07T09:06:09.2834811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2835276Z     outputs = self.model(
2025-09-07T09:06:09.2835633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2836011Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2836384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2836809Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2837155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2837526Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2837899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2838297Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2838687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.2839082Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.2839217Z 
2025-09-07T09:06:09.2839302Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2839501Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2839776Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2840010Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2840235Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2840577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2840892Z     return mod(**inputs)
2025-09-07T09:06:09.2841276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2841675Z     outputs = self.model(
2025-09-07T09:06:09.2842055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2842438Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2842819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2843200Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2843552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2843908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2844289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2844687Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2845080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2845486Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2845933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.2846422Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.2846614Z 
2025-09-07T09:06:09.2846717Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2847082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2847414Z     return mod(**inputs)
2025-09-07T09:06:09.2847773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2848165Z     outputs = self.model(
2025-09-07T09:06:09.2848522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2848909Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2849318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2849702Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2850052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2850456Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2850851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2851261Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2851658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2852094Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2852544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.2853013Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.2853172Z 
2025-09-07T09:06:09.2853275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2853631Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2853960Z     return mod(**inputs)
2025-09-07T09:06:09.2854359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2854748Z     outputs = self.model(
2025-09-07T09:06:09.2855102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2855526Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2855906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2856296Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2856637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2856998Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2857376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2857849Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2858288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.2858685Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.2858828Z 
2025-09-07T09:06:09.2858931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2859294Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2859621Z     return mod(**inputs)
2025-09-07T09:06:09.2859983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2860358Z     outputs = self.model(
2025-09-07T09:06:09.2860715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2861104Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2861482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2861852Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2862198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2862565Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2862946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.2863425Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.2863598Z 
2025-09-07T09:06:09.2863697Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2864054Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2864375Z     return mod(**inputs)
2025-09-07T09:06:09.2864771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2865148Z     outputs = self.model(
2025-09-07T09:06:09.2865493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2865868Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2866236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2866616Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2866954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2867308Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2867683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.2868138Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.2868517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.2868851Z     return self.act(input)
2025-09-07T09:06:09.2868964Z 
2025-09-07T09:06:09.2869062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2869472Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2869793Z     return mod(**inputs)
2025-09-07T09:06:09.2870137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2870515Z     outputs = self.model(
2025-09-07T09:06:09.2870859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2871242Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2871606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2871977Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2872320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2872686Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2873060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T09:06:09.2873449Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.2873583Z 
2025-09-07T09:06:09.2873686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2874028Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2874345Z     return mod(**inputs)
2025-09-07T09:06:09.2874689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2875074Z     outputs = self.model(
2025-09-07T09:06:09.2875425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2875793Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2876158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2876557Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2876895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2877296Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2877681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2878087Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2878501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.2878996Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.2879200Z 
2025-09-07T09:06:09.2879301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2879657Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2879978Z     return mod(**inputs)
2025-09-07T09:06:09.2880337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2880720Z     outputs = self.model(
2025-09-07T09:06:09.2881066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2881449Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2881857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2882247Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2882588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2882949Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2883359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2883763Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2884151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.2884534Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.2884668Z 
2025-09-07T09:06:09.2884763Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2885108Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2885431Z     return mod(**inputs)
2025-09-07T09:06:09.2885786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2886155Z     outputs = self.model(
2025-09-07T09:06:09.2886506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2886888Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2887257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2887629Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2887968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2888320Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2888695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2889085Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2889466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.2889855Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.2889996Z 
2025-09-07T09:06:09.2890074Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2890278Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2890467Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2890662Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2890923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2891295Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2891624Z     return mod(**inputs)
2025-09-07T09:06:09.2891987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2892419Z     outputs = self.model(
2025-09-07T09:06:09.2892780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2893167Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2893543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2893929Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2894273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2894641Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2895028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2895422Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2895856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2896269Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2896724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.2897306Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.2897535Z 
2025-09-07T09:06:09.2897646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2898004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2898328Z     return mod(**inputs)
2025-09-07T09:06:09.2898687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2899061Z     outputs = self.model(
2025-09-07T09:06:09.2899421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2899802Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2900169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2900544Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2900879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2901238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2901615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2902010Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2902393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2902797Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2903458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.2903926Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.2904084Z 
2025-09-07T09:06:09.2904194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2904541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2904864Z     return mod(**inputs)
2025-09-07T09:06:09.2905220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2905670Z     outputs = self.model(
2025-09-07T09:06:09.2906020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2906395Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2906825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2907206Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2907549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2907900Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2908277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2908674Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2909064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.2909448Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.2909574Z 
2025-09-07T09:06:09.2909669Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2910065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2910386Z     return mod(**inputs)
2025-09-07T09:06:09.2910740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2911117Z     outputs = self.model(
2025-09-07T09:06:09.2911508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2911893Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2912262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2912642Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2912975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2913329Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2913708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.2914130Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.2914299Z 
2025-09-07T09:06:09.2914401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2914737Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2915053Z     return mod(**inputs)
2025-09-07T09:06:09.2915397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2915777Z     outputs = self.model(
2025-09-07T09:06:09.2916123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2916497Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2916860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2917238Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2917576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2917921Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2918298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.2918715Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.2919137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.2919474Z     return self.act(input)
2025-09-07T09:06:09.2919578Z 
2025-09-07T09:06:09.2919672Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2920011Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2920371Z     return mod(**inputs)
2025-09-07T09:06:09.2920713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2921087Z     outputs = self.model(
2025-09-07T09:06:09.2921442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2921823Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2922189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2922561Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2922889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2923240Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2923660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T09:06:09.2924064Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.2924198Z 
2025-09-07T09:06:09.2924299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2924640Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2924961Z     return mod(**inputs)
2025-09-07T09:06:09.2926588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2926991Z     outputs = self.model(
2025-09-07T09:06:09.2927341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2927719Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2928080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2928459Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2928790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2929137Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2929506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2929900Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2930288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.2930739Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.2930940Z 
2025-09-07T09:06:09.2931034Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2931382Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2931704Z     return mod(**inputs)
2025-09-07T09:06:09.2932053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2932421Z     outputs = self.model(
2025-09-07T09:06:09.2932768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2933144Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2933505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2933934Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2934271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2934626Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2935006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2935449Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2935833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.2936212Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.2936347Z 
2025-09-07T09:06:09.2936444Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2936793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2937108Z     return mod(**inputs)
2025-09-07T09:06:09.2937532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2937918Z     outputs = self.model(
2025-09-07T09:06:09.2938273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2938699Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2939069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2939438Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2939797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2940200Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2940598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2941000Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2941400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.2941790Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.2941930Z 
2025-09-07T09:06:09.2942015Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2942221Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2942410Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2942602Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2942829Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2943187Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2943509Z     return mod(**inputs)
2025-09-07T09:06:09.2943869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2944254Z     outputs = self.model(
2025-09-07T09:06:09.2944612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2945001Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2945369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2945759Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2946114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2946479Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2946851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2947244Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2947633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2948090Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2948543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.2949023Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.2949258Z 
2025-09-07T09:06:09.2949360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2949723Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2950046Z     return mod(**inputs)
2025-09-07T09:06:09.2950408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2950783Z     outputs = self.model(
2025-09-07T09:06:09.2951136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2951520Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2951891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2952267Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2952648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2953015Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2953399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2953798Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2954215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2954622Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2955068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.2955533Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.2955691Z 
2025-09-07T09:06:09.2955797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2956145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2956468Z     return mod(**inputs)
2025-09-07T09:06:09.2956821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2957204Z     outputs = self.model(
2025-09-07T09:06:09.2957549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2957930Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2958304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2958691Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2959036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2959387Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2959778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2960169Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2960556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.2960939Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.2961070Z 
2025-09-07T09:06:09.2961170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2961515Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2961875Z     return mod(**inputs)
2025-09-07T09:06:09.2962230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2962600Z     outputs = self.model(
2025-09-07T09:06:09.2962948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2963369Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2963735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2964108Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2964440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2964790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2965163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.2965586Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.2965756Z 
2025-09-07T09:06:09.2965858Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2966208Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2966610Z     return mod(**inputs)
2025-09-07T09:06:09.2966980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2967364Z     outputs = self.model(
2025-09-07T09:06:09.2967704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2968119Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2968487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2968865Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2969200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2969547Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2969925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.2970341Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.2970722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.2971057Z     return self.act(input)
2025-09-07T09:06:09.2971167Z 
2025-09-07T09:06:09.2971263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2971606Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2971923Z     return mod(**inputs)
2025-09-07T09:06:09.2972274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2972649Z     outputs = self.model(
2025-09-07T09:06:09.2972997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2973378Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2973741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2974119Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2974448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2974800Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2975173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T09:06:09.2975601Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.2975734Z 
2025-09-07T09:06:09.2975829Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2976173Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2976484Z     return mod(**inputs)
2025-09-07T09:06:09.2976884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2977264Z     outputs = self.model(
2025-09-07T09:06:09.2977659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2978038Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2978407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2978781Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2979104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2979455Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2979826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2980263Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2980655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.2981101Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.2981307Z 
2025-09-07T09:06:09.2981402Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2981782Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2982105Z     return mod(**inputs)
2025-09-07T09:06:09.2982456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2982824Z     outputs = self.model(
2025-09-07T09:06:09.2983171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2983548Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2983912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2984281Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2984610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2984959Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2985336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2985726Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2986103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.2986486Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.2986619Z 
2025-09-07T09:06:09.2986740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2987085Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2987398Z     return mod(**inputs)
2025-09-07T09:06:09.2987734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2988105Z     outputs = self.model(
2025-09-07T09:06:09.2988452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2988827Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2989221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2989597Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2989930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2990283Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2990686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2991073Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2991462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.2991852Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.2991989Z 
2025-09-07T09:06:09.2992073Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2992271Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2992471Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2992663Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.2992880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2993219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2993537Z     return mod(**inputs)
2025-09-07T09:06:09.2993923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.2994302Z     outputs = self.model(
2025-09-07T09:06:09.2994649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.2995021Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.2995415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.2995800Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.2996156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.2996527Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.2996891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.2997285Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.2997671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.2998070Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.2998510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.2998987Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.2999177Z 
2025-09-07T09:06:09.2999271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.2999614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.2999929Z     return mod(**inputs)
2025-09-07T09:06:09.3000272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3000652Z     outputs = self.model(
2025-09-07T09:06:09.3000997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3001376Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3001743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3002113Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3002446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3002841Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3003403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3003795Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3004191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3004655Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3005103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3005561Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3005723Z 
2025-09-07T09:06:09.3005837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3006192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3006519Z     return mod(**inputs)
2025-09-07T09:06:09.3006887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3007269Z     outputs = self.model(
2025-09-07T09:06:09.3007663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3008055Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3008431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3008813Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3009198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3009563Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3009950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3010346Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3010732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3011109Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3011248Z 
2025-09-07T09:06:09.3011345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3011697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3012011Z     return mod(**inputs)
2025-09-07T09:06:09.3012364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3012734Z     outputs = self.model(
2025-09-07T09:06:09.3013082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3013461Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3013827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3014199Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3014539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3014896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3015270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.3015689Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3015856Z 
2025-09-07T09:06:09.3015953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3016295Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3016614Z     return mod(**inputs)
2025-09-07T09:06:09.3017019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3017397Z     outputs = self.model(
2025-09-07T09:06:09.3017807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3018231Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3018605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3018988Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3019314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3019677Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3020058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.3020479Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3020857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3021186Z     return self.act(input)
2025-09-07T09:06:09.3021298Z 
2025-09-07T09:06:09.3021389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3021777Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3022100Z     return mod(**inputs)
2025-09-07T09:06:09.3022449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3022829Z     outputs = self.model(
2025-09-07T09:06:09.3023219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3023606Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3023983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3024353Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3024695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3025058Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3025441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T09:06:09.3025827Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3025957Z 
2025-09-07T09:06:09.3026054Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3026414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3026736Z     return mod(**inputs)
2025-09-07T09:06:09.3027096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3027468Z     outputs = self.model(
2025-09-07T09:06:09.3027819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3028198Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3028570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3028949Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3029283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3029640Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3030022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3030418Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3030875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3031317Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3031528Z 
2025-09-07T09:06:09.3031627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3032018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3032340Z     return mod(**inputs)
2025-09-07T09:06:09.3032686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3033069Z     outputs = self.model(
2025-09-07T09:06:09.3033421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3033801Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3034164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3034539Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3034877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3035233Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3035645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3036045Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3036425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3036805Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3036969Z 
2025-09-07T09:06:09.3037075Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3037422Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3037736Z     return mod(**inputs)
2025-09-07T09:06:09.3038088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3038463Z     outputs = self.model(
2025-09-07T09:06:09.3038817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3039207Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3039564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3039934Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3040273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3040625Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3040990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3041377Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3041765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3042159Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3042299Z 
2025-09-07T09:06:09.3042383Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3042583Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3042783Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3042979Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3043201Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3043540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3043855Z     return mod(**inputs)
2025-09-07T09:06:09.3044203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3044633Z     outputs = self.model(
2025-09-07T09:06:09.3044985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3045359Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3045768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3046145Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3046480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3046829Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3047199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3047591Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3047976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3048376Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3048818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3049336Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3049527Z 
2025-09-07T09:06:09.3049623Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3049971Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3050291Z     return mod(**inputs)
2025-09-07T09:06:09.3050674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3051069Z     outputs = self.model(
2025-09-07T09:06:09.3051419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3051799Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3052157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3052538Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3052874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3053229Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3053601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3053986Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3054367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3054772Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3055217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3055672Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3055833Z 
2025-09-07T09:06:09.3055928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3056273Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3056587Z     return mod(**inputs)
2025-09-07T09:06:09.3056941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3057311Z     outputs = self.model(
2025-09-07T09:06:09.3057705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3058130Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3058501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3058877Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3059204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3059605Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3059981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3060371Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3060759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3061140Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3061273Z 
2025-09-07T09:06:09.3061369Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3061717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3062031Z     return mod(**inputs)
2025-09-07T09:06:09.3062372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3062752Z     outputs = self.model(
2025-09-07T09:06:09.3063131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3063517Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3063879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3064284Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3064620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3064976Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3065351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.3065791Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3065958Z 
2025-09-07T09:06:09.3066049Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3066394Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3066709Z     return mod(**inputs)
2025-09-07T09:06:09.3067054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3067423Z     outputs = self.model(
2025-09-07T09:06:09.3067776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3068154Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3068520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3068892Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3069219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3069572Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3069948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.3070358Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3070726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3071067Z     return self.act(input)
2025-09-07T09:06:09.3071179Z 
2025-09-07T09:06:09.3071272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3071624Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3071980Z     return mod(**inputs)
2025-09-07T09:06:09.3072321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3072704Z     outputs = self.model(
2025-09-07T09:06:09.3073055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3073477Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3073846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3074219Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3085269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3085711Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3086135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T09:06:09.3086542Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3086683Z 
2025-09-07T09:06:09.3086791Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3087233Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3087574Z     return mod(**inputs)
2025-09-07T09:06:09.3087960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3088348Z     outputs = self.model(
2025-09-07T09:06:09.3088755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3089149Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3089519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3089901Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3090251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3090618Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3090999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3091402Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3091794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3092258Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3092466Z 
2025-09-07T09:06:09.3092574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3092934Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3093248Z     return mod(**inputs)
2025-09-07T09:06:09.3093599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3093978Z     outputs = self.model(
2025-09-07T09:06:09.3094328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3094698Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3095059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3095431Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3095774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3096128Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3096494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3096929Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3097310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3097766Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3097930Z 
2025-09-07T09:06:09.3098028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3098370Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3098680Z     return mod(**inputs)
2025-09-07T09:06:09.3099024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3099393Z     outputs = self.model(
2025-09-07T09:06:09.3099733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3100109Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3100464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3100834Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3101200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3101560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3101934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3102334Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3102752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3103307Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3103447Z 
2025-09-07T09:06:09.3103526Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3103727Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3103926Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3104114Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3104322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3104668Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3104985Z     return mod(**inputs)
2025-09-07T09:06:09.3105330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3105694Z     outputs = self.model(
2025-09-07T09:06:09.3106041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3106409Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3106767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3107133Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3107467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3107816Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3108188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3108577Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3108950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3109345Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3109790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3110268Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3110601Z 
2025-09-07T09:06:09.3110703Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3111040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3111349Z     return mod(**inputs)
2025-09-07T09:06:09.3111746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3112119Z     outputs = self.model(
2025-09-07T09:06:09.3112455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3112825Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3113182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3113558Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3113892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3114232Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3114606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3114997Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3115605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3116022Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3116461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3116992Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3117158Z 
2025-09-07T09:06:09.3117254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3117599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3117912Z     return mod(**inputs)
2025-09-07T09:06:09.3118250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3118638Z     outputs = self.model(
2025-09-07T09:06:09.3118985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3119353Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3119703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3120069Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3120402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3120744Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3121112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward
2025-09-07T09:06:09.3121487Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:06:09.3121866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3122244Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3122368Z 
2025-09-07T09:06:09.3122465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3122801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3123102Z     return mod(**inputs)
2025-09-07T09:06:09.3123439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3123805Z     outputs = self.model(
2025-09-07T09:06:09.3124144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3124567Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3124925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3125297Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3125670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3126019Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3126382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.3126803Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3126973Z 
2025-09-07T09:06:09.3127066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3127401Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3127705Z     return mod(**inputs)
2025-09-07T09:06:09.3128056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3128424Z     outputs = self.model(
2025-09-07T09:06:09.3128802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3129185Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3129560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3129934Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3130308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3130695Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3131064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward
2025-09-07T09:06:09.3131472Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3131867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3132202Z     return self.act(input)
2025-09-07T09:06:09.3132316Z 
2025-09-07T09:06:09.3132430Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3132767Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3133094Z     return mod(**inputs)
2025-09-07T09:06:09.3133438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3133808Z     outputs = self.model(
2025-09-07T09:06:09.3134162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward
2025-09-07T09:06:09.3134543Z     encoder_outputs = self.encoder(
2025-09-07T09:06:09.3134907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward
2025-09-07T09:06:09.3135280Z     layer_outputs = encoder_layer(
2025-09-07T09:06:09.3135608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3135958Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3136332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward
2025-09-07T09:06:09.3136708Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3136839Z 
2025-09-07T09:06:09.3137092Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3137431Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3137844Z     return mod(**inputs)
2025-09-07T09:06:09.3138192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3138592Z     outputs = self.model(
2025-09-07T09:06:09.3138935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3139371Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3139740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3140114Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3140449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3140821Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3141209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3141617Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3142011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3142458Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3142667Z 
2025-09-07T09:06:09.3142802Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3143145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3143457Z     return mod(**inputs)
2025-09-07T09:06:09.3143805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3144212Z     outputs = self.model(
2025-09-07T09:06:09.3144559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3144935Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3145296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3145667Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3145998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3146349Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3146717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3147108Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3147496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3147867Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3147996Z 
2025-09-07T09:06:09.3148089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3148423Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3148734Z     return mod(**inputs)
2025-09-07T09:06:09.3149071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3149436Z     outputs = self.model(
2025-09-07T09:06:09.3149786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3150164Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3150517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3150886Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3151210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3151604Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3151981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3152374Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3152762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3153181Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3153319Z 
2025-09-07T09:06:09.3153395Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3153593Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3153785Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3153975Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3154188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3154520Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3154828Z     return mod(**inputs)
2025-09-07T09:06:09.3155173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3155538Z     outputs = self.model(
2025-09-07T09:06:09.3155909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3156285Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3156643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3157009Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3157366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3157715Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3158082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3158468Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3158854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3159240Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3159682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3160155Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3160333Z 
2025-09-07T09:06:09.3160425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3160757Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3161058Z     return mod(**inputs)
2025-09-07T09:06:09.3161394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3161767Z     outputs = self.model(
2025-09-07T09:06:09.3162098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3162464Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3162818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3163184Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3163513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3163860Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3164225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3164613Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3165040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3165421Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3165849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3166341Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3166497Z 
2025-09-07T09:06:09.3166592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3166931Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3167231Z     return mod(**inputs)
2025-09-07T09:06:09.3167567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3167935Z     outputs = self.model(
2025-09-07T09:06:09.3168273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3168646Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3169004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3169372Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3169746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3170125Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3170516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3170951Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3171344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3171723Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3171853Z 
2025-09-07T09:06:09.3171950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3172289Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3172598Z     return mod(**inputs)
2025-09-07T09:06:09.3172941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3173305Z     outputs = self.model(
2025-09-07T09:06:09.3173641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3174013Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3174374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3174745Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3175074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3175420Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3175787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3176193Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3176593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3177037Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3177236Z 
2025-09-07T09:06:09.3177328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3177741Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3178053Z     return mod(**inputs)
2025-09-07T09:06:09.3178401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3178828Z     outputs = self.model(
2025-09-07T09:06:09.3179173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3179551Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3179960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3180339Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3180673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3181024Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3181400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3181802Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3182201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3182571Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3182697Z 
2025-09-07T09:06:09.3182793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3183167Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3183476Z     return mod(**inputs)
2025-09-07T09:06:09.3183817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3184183Z     outputs = self.model(
2025-09-07T09:06:09.3184573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3184952Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3185321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3185687Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3186020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3186365Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3186747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3187149Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3187546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3187929Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3188061Z 
2025-09-07T09:06:09.3188137Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3188330Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3188513Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3188695Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3188905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3189245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3189549Z     return mod(**inputs)
2025-09-07T09:06:09.3189900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3190269Z     outputs = self.model(
2025-09-07T09:06:09.3190613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3190987Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3191346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3191759Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3192089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3192440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3192805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3193249Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3193646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3194038Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3194477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3194945Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3195129Z 
2025-09-07T09:06:09.3195221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3195558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3195872Z     return mod(**inputs)
2025-09-07T09:06:09.3196217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3196627Z     outputs = self.model(
2025-09-07T09:06:09.3196970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3197340Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3197696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3198099Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3198433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3198788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3199160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3199563Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3199957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3200342Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3200769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3201217Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3201368Z 
2025-09-07T09:06:09.3201465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3201800Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3202109Z     return mod(**inputs)
2025-09-07T09:06:09.3202445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3202505Z     outputs = self.model(
2025-09-07T09:06:09.3202757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3202823Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3203220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3203286Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3203506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3203581Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3203825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3204011Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3204255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3204326Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3204378Z 
2025-09-07T09:06:09.3204470Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3204654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3204716Z     return mod(**inputs)
2025-09-07T09:06:09.3204959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3205022Z     outputs = self.model(
2025-09-07T09:06:09.3205265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3205329Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3205583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3205646Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3205904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3205973Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3206216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3206334Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3206338Z 
2025-09-07T09:06:09.3206473Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3206662Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3206720Z     return mod(**inputs)
2025-09-07T09:06:09.3206971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3207029Z     outputs = self.model(
2025-09-07T09:06:09.3207272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3207339Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3207583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3207651Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3207864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3207930Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3208174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3208277Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3208490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3208551Z     return self.act(input)
2025-09-07T09:06:09.3208554Z 
2025-09-07T09:06:09.3208645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3208835Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3208891Z     return mod(**inputs)
2025-09-07T09:06:09.3209140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3209195Z     outputs = self.model(
2025-09-07T09:06:09.3209442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3209502Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3209791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3209859Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3210073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3210185Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3210429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T09:06:09.3210501Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3210505Z 
2025-09-07T09:06:09.3210598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3210789Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3210852Z     return mod(**inputs)
2025-09-07T09:06:09.3211096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3211154Z     outputs = self.model(
2025-09-07T09:06:09.3211403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3211468Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3211752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3211817Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3212033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3212103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3212374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3212467Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3212712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3212857Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3212861Z 
2025-09-07T09:06:09.3212951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3213140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3213201Z     return mod(**inputs)
2025-09-07T09:06:09.3213442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3213506Z     outputs = self.model(
2025-09-07T09:06:09.3213751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3213818Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3214059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3214122Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3214339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3214409Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3214652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3214740Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3214981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3215053Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3215056Z 
2025-09-07T09:06:09.3215144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3215372Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3215427Z     return mod(**inputs)
2025-09-07T09:06:09.3215670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3215731Z     outputs = self.model(
2025-09-07T09:06:09.3216011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3216077Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3216322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3216389Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3216603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3216669Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3216913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3216996Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3217240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3217362Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3217366Z 
2025-09-07T09:06:09.3217439Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3217575Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3217644Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3217713Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3217844Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3218032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3218093Z     return mod(**inputs)
2025-09-07T09:06:09.3218337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3218399Z     outputs = self.model(
2025-09-07T09:06:09.3218641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3218707Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3218953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3219014Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3219230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3219297Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3219541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3219628Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3219871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3219957Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3220252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3220380Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3220385Z 
2025-09-07T09:06:09.3220474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3220658Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3220719Z     return mod(**inputs)
2025-09-07T09:06:09.3220970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3221083Z     outputs = self.model(
2025-09-07T09:06:09.3221330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3221398Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3221643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3221743Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3221962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3222030Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3222273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3222357Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3222595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3222685Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3222974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3223073Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3223114Z 
2025-09-07T09:06:09.3223206Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3223393Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3223448Z     return mod(**inputs)
2025-09-07T09:06:09.3223734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3223798Z     outputs = self.model(
2025-09-07T09:06:09.3224042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3224111Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3224352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3224415Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3224634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3224702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3224947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3225029Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3225270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3225344Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3225349Z 
2025-09-07T09:06:09.3225435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3225627Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3225682Z     return mod(**inputs)
2025-09-07T09:06:09.3225929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3225987Z     outputs = self.model(
2025-09-07T09:06:09.3226230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3226295Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3226538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3226604Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3226818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3226921Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3227164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3227260Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3227543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3227685Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3227689Z 
2025-09-07T09:06:09.3227784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3227973Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3228028Z     return mod(**inputs)
2025-09-07T09:06:09.3228275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3228333Z     outputs = self.model(
2025-09-07T09:06:09.3228580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3228641Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3228921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3228992Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3229207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3229275Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3229547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3229648Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3229893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3229962Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3229965Z 
2025-09-07T09:06:09.3230056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3230241Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3230303Z     return mod(**inputs)
2025-09-07T09:06:09.3230545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3230603Z     outputs = self.model(
2025-09-07T09:06:09.3230852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3230913Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3231157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3231222Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3231432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3231504Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3231743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3231841Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3232082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3232160Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3232166Z 
2025-09-07T09:06:09.3232238Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3232306Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3232380Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3232487Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3232577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3232765Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3232820Z     return mod(**inputs)
2025-09-07T09:06:09.3233105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3233163Z     outputs = self.model(
2025-09-07T09:06:09.3233411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3233475Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3233720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3233786Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3233996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3234069Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3234307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3234399Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3234675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3234762Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3235056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3235208Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3235212Z 
2025-09-07T09:06:09.3235307Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3235493Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3235551Z     return mod(**inputs)
2025-09-07T09:06:09.3235798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3235857Z     outputs = self.model(
2025-09-07T09:06:09.3236101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3236165Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3236406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3236471Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3236685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3236755Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3236995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3237088Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3237331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3237415Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3237707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3237801Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3237804Z 
2025-09-07T09:06:09.3237899Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3238082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3238177Z     return mod(**inputs)
2025-09-07T09:06:09.3238425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3238484Z     outputs = self.model(
2025-09-07T09:06:09.3238732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3238862Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3239111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3239178Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3239391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3239462Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3239699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3239797Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3240037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3240106Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3240111Z 
2025-09-07T09:06:09.3240237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3240424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3240483Z     return mod(**inputs)
2025-09-07T09:06:09.3240729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3240787Z     outputs = self.model(
2025-09-07T09:06:09.3241066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3241131Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3241382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3241443Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3241657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3241729Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3241971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3242084Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3242088Z 
2025-09-07T09:06:09.3242178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3242364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3242421Z     return mod(**inputs)
2025-09-07T09:06:09.3242664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3242723Z     outputs = self.model(
2025-09-07T09:06:09.3242964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3243032Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3243275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3243336Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3243551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3243619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3243860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3243997Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3244205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3244276Z     return self.act(input)
2025-09-07T09:06:09.3244280Z 
2025-09-07T09:06:09.3244378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3244614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3244675Z     return mod(**inputs)
2025-09-07T09:06:09.3244933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3244997Z     outputs = self.model(
2025-09-07T09:06:09.3245244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3245318Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3245562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3245633Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3245853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3245926Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3246214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T09:06:09.3246288Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3246292Z 
2025-09-07T09:06:09.3246393Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3246613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3246677Z     return mod(**inputs)
2025-09-07T09:06:09.3246925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3246985Z     outputs = self.model(
2025-09-07T09:06:09.3247229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3247295Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3247546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3247613Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3247833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3247907Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3248151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3248248Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3248489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3248635Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3248641Z 
2025-09-07T09:06:09.3248734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3248925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3248987Z     return mod(**inputs)
2025-09-07T09:06:09.3249230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3249294Z     outputs = self.model(
2025-09-07T09:06:09.3249537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3249603Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3249849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3249950Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3250168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3250236Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3250517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3250610Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3250852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3250928Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3250933Z 
2025-09-07T09:06:09.3251023Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3251215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3251276Z     return mod(**inputs)
2025-09-07T09:06:09.3251519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3251582Z     outputs = self.model(
2025-09-07T09:06:09.3251859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3251930Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3252173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3252241Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3252551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3252618Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3252869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3252955Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3253196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3253276Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3253282Z 
2025-09-07T09:06:09.3253355Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3253427Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3253498Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3253564Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3253659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3253845Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3253904Z     return mod(**inputs)
2025-09-07T09:06:09.3254148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3254212Z     outputs = self.model(
2025-09-07T09:06:09.3254454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3254516Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3254767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3254828Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3255050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3255118Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3255357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3255441Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3255727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3255819Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3256117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3256277Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3256284Z 
2025-09-07T09:06:09.3256378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3256566Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3256625Z     return mod(**inputs)
2025-09-07T09:06:09.3256873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3256935Z     outputs = self.model(
2025-09-07T09:06:09.3257180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3257242Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3257529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3257628Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3257849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3257916Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3258162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3258286Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3258542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3258636Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3258935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3259051Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3259057Z 
2025-09-07T09:06:09.3259158Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3259348Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3259410Z     return mod(**inputs)
2025-09-07T09:06:09.3259662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3259726Z     outputs = self.model(
2025-09-07T09:06:09.3259969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3260035Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3260287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3260354Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3260574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3260652Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3260897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3261017Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3261259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3261335Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3261338Z 
2025-09-07T09:06:09.3261429Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3261673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3261752Z     return mod(**inputs)
2025-09-07T09:06:09.3262001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3262097Z     outputs = self.model(
2025-09-07T09:06:09.3262344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3262410Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3262654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3262719Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3262935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3263003Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3263272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3263372Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3263649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3263795Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3263798Z 
2025-09-07T09:06:09.3263888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3264104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3264217Z     return mod(**inputs)
2025-09-07T09:06:09.3264474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3264538Z     outputs = self.model(
2025-09-07T09:06:09.3264783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3264851Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3265096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3265169Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3265383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3265452Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3265697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3265795Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3266044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3266116Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3266119Z 
2025-09-07T09:06:09.3266212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3266397Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3266458Z     return mod(**inputs)
2025-09-07T09:06:09.3266707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3266766Z     outputs = self.model(
2025-09-07T09:06:09.3267014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3267078Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3267321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3267423Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3267635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3267704Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3267948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3268081Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3268324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3268399Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3268402Z 
2025-09-07T09:06:09.3268481Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3268550Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3268621Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3268688Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3268782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3268967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3269024Z     return mod(**inputs)
2025-09-07T09:06:09.3269308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3269370Z     outputs = self.model(
2025-09-07T09:06:09.3269613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3269682Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3269954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3270024Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3270237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3270309Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3270551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3270643Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3270887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3270974Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3271271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3271395Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3271398Z 
2025-09-07T09:06:09.3271488Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3271679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3271739Z     return mod(**inputs)
2025-09-07T09:06:09.3271989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3272048Z     outputs = self.model(
2025-09-07T09:06:09.3272289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3272356Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3272598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3272668Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3272882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3272959Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3273197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3273324Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3273569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3273684Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3273976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3274071Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3274074Z 
2025-09-07T09:06:09.3274165Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3274357Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3274414Z     return mod(**inputs)
2025-09-07T09:06:09.3274665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3274725Z     outputs = self.model(
2025-09-07T09:06:09.3274969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3275043Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3275322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3275399Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3275610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3275681Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3275962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3276057Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3276303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3276375Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3276378Z 
2025-09-07T09:06:09.3276469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3276655Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3276711Z     return mod(**inputs)
2025-09-07T09:06:09.3276958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3277016Z     outputs = self.model(
2025-09-07T09:06:09.3277265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3277327Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3277574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3277639Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3277849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3277916Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3278159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3278270Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3278273Z 
2025-09-07T09:06:09.3278362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3278548Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3278606Z     return mod(**inputs)
2025-09-07T09:06:09.3278851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3278947Z     outputs = self.model(
2025-09-07T09:06:09.3279190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3279252Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3279538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3279606Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3279820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3279888Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3280133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3280238Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3280448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3280511Z     return self.act(input)
2025-09-07T09:06:09.3280514Z 
2025-09-07T09:06:09.3280604Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3280824Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3280885Z     return mod(**inputs)
2025-09-07T09:06:09.3281132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3281193Z     outputs = self.model(
2025-09-07T09:06:09.3281483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3281551Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3281794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3281862Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3282075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3282143Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3282388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T09:06:09.3282463Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3282466Z 
2025-09-07T09:06:09.3282559Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3282741Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3282796Z     return mod(**inputs)
2025-09-07T09:06:09.3283049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3283107Z     outputs = self.model(
2025-09-07T09:06:09.3283351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3283412Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3283655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3283722Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3283935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3284003Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3284244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3284334Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3284580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3284758Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3284761Z 
2025-09-07T09:06:09.3284856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3285040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3285163Z     return mod(**inputs)
2025-09-07T09:06:09.3285408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3285468Z     outputs = self.model(
2025-09-07T09:06:09.3285715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3285779Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3286026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3286091Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3286303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3286371Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3286644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3286740Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3286984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3287057Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3287061Z 
2025-09-07T09:06:09.3287180Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3287366Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3287430Z     return mod(**inputs)
2025-09-07T09:06:09.3287672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3287732Z     outputs = self.model(
2025-09-07T09:06:09.3287976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3288043Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3288289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3288352Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3288588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3288660Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3288901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3288990Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3289228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3289309Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3289313Z 
2025-09-07T09:06:09.3289388Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3289463Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3289529Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3289596Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3289688Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3289868Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3289926Z     return mod(**inputs)
2025-09-07T09:06:09.3290170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3290262Z     outputs = self.model(
2025-09-07T09:06:09.3290508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3290571Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3290819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3290915Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3291127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3291196Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3291438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3291528Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3291769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3291858Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3292151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3292310Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3292313Z 
2025-09-07T09:06:09.3292407Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3292591Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3292652Z     return mod(**inputs)
2025-09-07T09:06:09.3292931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3292996Z     outputs = self.model(
2025-09-07T09:06:09.3293242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3293306Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3293553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3293618Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3293834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3293903Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3294142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3294229Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3294470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3294554Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3294840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3294935Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3294939Z 
2025-09-07T09:06:09.3295034Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3295218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3295281Z     return mod(**inputs)
2025-09-07T09:06:09.3295524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3295583Z     outputs = self.model(
2025-09-07T09:06:09.3295828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3295890Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3296187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3296250Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3296464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3296533Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3296820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3296908Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3297149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3297223Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3297226Z 
2025-09-07T09:06:09.3297317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3297540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3297602Z     return mod(**inputs)
2025-09-07T09:06:09.3297847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3297911Z     outputs = self.model(
2025-09-07T09:06:09.3298196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3298262Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3298510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3298574Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3298824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3298891Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3299138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3299234Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3299476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3299622Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3299625Z 
2025-09-07T09:06:09.3299714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3299901Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3299957Z     return mod(**inputs)
2025-09-07T09:06:09.3300201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3300265Z     outputs = self.model(
2025-09-07T09:06:09.3300508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3300581Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3300822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3300898Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3301115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3301183Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3301430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3301523Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3301767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3301883Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3301886Z 
2025-09-07T09:06:09.3301977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3302165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3302222Z     return mod(**inputs)
2025-09-07T09:06:09.3302513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3302576Z     outputs = self.model(
2025-09-07T09:06:09.3302822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3302889Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3303275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3303348Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3303565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3303635Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3303884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3304093Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3304347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3304424Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3304428Z 
2025-09-07T09:06:09.3304507Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3304577Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3304698Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3304772Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3304868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3305061Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3305118Z     return mod(**inputs)
2025-09-07T09:06:09.3305368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3305433Z     outputs = self.model(
2025-09-07T09:06:09.3305679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3305749Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3305990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3306055Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3306280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3306352Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3306596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3306689Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3306938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3307028Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3307322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3307445Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3307448Z 
2025-09-07T09:06:09.3307543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3307734Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3307866Z     return mod(**inputs)
2025-09-07T09:06:09.3308115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3308180Z     outputs = self.model(
2025-09-07T09:06:09.3308421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3308538Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3308785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3308850Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3309064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3309132Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3309381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3309474Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3309718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3309802Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3310126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3310226Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3310229Z 
2025-09-07T09:06:09.3310319Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3310539Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3310596Z     return mod(**inputs)
2025-09-07T09:06:09.3310843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3310905Z     outputs = self.model(
2025-09-07T09:06:09.3311149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3311216Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3311459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3311526Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3311739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3311808Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3312053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3312144Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3312391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3312462Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3312465Z 
2025-09-07T09:06:09.3312557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3312741Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3312799Z     return mod(**inputs)
2025-09-07T09:06:09.3313047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3313104Z     outputs = self.model(
2025-09-07T09:06:09.3313352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3313416Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3313660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3314018Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3314236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3314306Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3314547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3314698Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3314702Z 
2025-09-07T09:06:09.3314795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3314978Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3315041Z     return mod(**inputs)
2025-09-07T09:06:09.3315283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3315346Z     outputs = self.model(
2025-09-07T09:06:09.3315603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3315668Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3315959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3316024Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3316244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3316312Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3316588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3316695Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3316905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3316972Z     return self.act(input)
2025-09-07T09:06:09.3316975Z 
2025-09-07T09:06:09.3317063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3317246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3317306Z     return mod(**inputs)
2025-09-07T09:06:09.3317550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3317609Z     outputs = self.model(
2025-09-07T09:06:09.3317853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3317924Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3318171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3318237Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3318457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3318525Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3318773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T09:06:09.3318851Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3318854Z 
2025-09-07T09:06:09.3318946Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3319138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3319196Z     return mod(**inputs)
2025-09-07T09:06:09.3319448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3319509Z     outputs = self.model(
2025-09-07T09:06:09.3319756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3319877Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3320124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3320198Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3320453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3320529Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3320775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3320869Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3321127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3321273Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3321277Z 
2025-09-07T09:06:09.3321379Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3321565Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3321625Z     return mod(**inputs)
2025-09-07T09:06:09.3321914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3321976Z     outputs = self.model(
2025-09-07T09:06:09.3322229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3322294Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3322578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3322645Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3322863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3322938Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3323182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3323284Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3323527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3323600Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3323603Z 
2025-09-07T09:06:09.3323700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3323885Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3330744Z     return mod(**inputs)
2025-09-07T09:06:09.3331069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3331149Z     outputs = self.model(
2025-09-07T09:06:09.3331433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3331508Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3331779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3331847Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3332087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3332164Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3332421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3332525Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3332858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3332942Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3332947Z 
2025-09-07T09:06:09.3333026Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3333156Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3333226Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3333294Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3333401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3333610Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3333674Z     return mod(**inputs)
2025-09-07T09:06:09.3333945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3334011Z     outputs = self.model(
2025-09-07T09:06:09.3334265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3334333Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3334583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3334687Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3334912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3334994Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3335240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3335368Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3335615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3335707Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3336007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3336136Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3336143Z 
2025-09-07T09:06:09.3336244Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3336440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3336502Z     return mod(**inputs)
2025-09-07T09:06:09.3336752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3336815Z     outputs = self.model(
2025-09-07T09:06:09.3337063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3337131Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3337376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3337439Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3337730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3337809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3338051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3338143Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3338384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3338479Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3338768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3338914Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3338918Z 
2025-09-07T09:06:09.3339014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3339205Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3339297Z     return mod(**inputs)
2025-09-07T09:06:09.3339541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3339600Z     outputs = self.model(
2025-09-07T09:06:09.3339849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3339914Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3340158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3340223Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3340438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3340509Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3340782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3340874Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3341115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3341188Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3341191Z 
2025-09-07T09:06:09.3341311Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3341501Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3341562Z     return mod(**inputs)
2025-09-07T09:06:09.3341813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3341874Z     outputs = self.model(
2025-09-07T09:06:09.3342117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3342182Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3342426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3342489Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3342704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3342769Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3343014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3343119Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3343362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3343517Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3343523Z 
2025-09-07T09:06:09.3343617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3343811Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3343868Z     return mod(**inputs)
2025-09-07T09:06:09.3344116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3344180Z     outputs = self.model(
2025-09-07T09:06:09.3344422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3344527Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3344773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3344839Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3345059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3345161Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3345409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3345506Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3345751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3345821Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3345826Z 
2025-09-07T09:06:09.3345917Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3346113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3346175Z     return mod(**inputs)
2025-09-07T09:06:09.3346457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3346517Z     outputs = self.model(
2025-09-07T09:06:09.3346762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3346833Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3347110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3347182Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3347399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3347475Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3347716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3347806Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3348058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3348137Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3348140Z 
2025-09-07T09:06:09.3348220Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3348289Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3348358Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3348434Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3348525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3348712Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3348773Z     return mod(**inputs)
2025-09-07T09:06:09.3349016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3349082Z     outputs = self.model(
2025-09-07T09:06:09.3349326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3349396Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3349641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3349705Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3349925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3349993Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3350321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3350421Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3350675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3350801Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3351095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3351226Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3351230Z 
2025-09-07T09:06:09.3351322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3351512Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3351568Z     return mod(**inputs)
2025-09-07T09:06:09.3351813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3351877Z     outputs = self.model(
2025-09-07T09:06:09.3352120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3352188Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3352526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3352593Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3352807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3352904Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3353153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3353249Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3353492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3353579Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3353870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3353972Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3353976Z 
2025-09-07T09:06:09.3354069Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3354266Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3354326Z     return mod(**inputs)
2025-09-07T09:06:09.3354578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3354639Z     outputs = self.model(
2025-09-07T09:06:09.3354882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3354951Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3355193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3355264Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3355481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3355550Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3355794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3355889Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3356131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3356251Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3356255Z 
2025-09-07T09:06:09.3356347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3356533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3356623Z     return mod(**inputs)
2025-09-07T09:06:09.3356876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3356936Z     outputs = self.model(
2025-09-07T09:06:09.3357180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3357245Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3357489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3357560Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3357772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3357839Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3358082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3358231Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3358235Z 
2025-09-07T09:06:09.3358331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3358515Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3358576Z     return mod(**inputs)
2025-09-07T09:06:09.3358851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3358911Z     outputs = self.model(
2025-09-07T09:06:09.3359164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3359229Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3359474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3359540Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3359759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3359826Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3360068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3360176Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3360386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3360454Z     return self.act(input)
2025-09-07T09:06:09.3360458Z 
2025-09-07T09:06:09.3360547Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3360732Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3360794Z     return mod(**inputs)
2025-09-07T09:06:09.3361039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3361100Z     outputs = self.model(
2025-09-07T09:06:09.3361343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3361406Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3361655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3361716Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3361972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3362042Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3362293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T09:06:09.3362402Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3362407Z 
2025-09-07T09:06:09.3362502Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3362696Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3362755Z     return mod(**inputs)
2025-09-07T09:06:09.3363012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3363071Z     outputs = self.model(
2025-09-07T09:06:09.3363326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3363397Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3363638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3363704Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3363953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3364028Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3364277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3364367Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3364646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3364791Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3364796Z 
2025-09-07T09:06:09.3364890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3365075Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3365134Z     return mod(**inputs)
2025-09-07T09:06:09.3365384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3365443Z     outputs = self.model(
2025-09-07T09:06:09.3365688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3365752Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3365994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3366060Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3366272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3366346Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3366587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3366675Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3366918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3366988Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3366991Z 
2025-09-07T09:06:09.3367084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3367267Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3367324Z     return mod(**inputs)
2025-09-07T09:06:09.3367567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3367659Z     outputs = self.model(
2025-09-07T09:06:09.3367905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3367969Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3368216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3368310Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3368522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3368589Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3368830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3368915Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3369153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3369233Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3369236Z 
2025-09-07T09:06:09.3369308Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3369375Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3369445Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3369543Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3369635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3369818Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3369874Z     return mod(**inputs)
2025-09-07T09:06:09.3370171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3370231Z     outputs = self.model(
2025-09-07T09:06:09.3370492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3370557Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3370798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3370863Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3371076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3371144Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3371383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3371467Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3371708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3371794Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3372096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3372226Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3372229Z 
2025-09-07T09:06:09.3372327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3372514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3372573Z     return mod(**inputs)
2025-09-07T09:06:09.3372823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3372880Z     outputs = self.model(
2025-09-07T09:06:09.3373131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3373192Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3373475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3373548Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3373760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3373867Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3378649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3378758Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3379034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3379135Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3379443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3379549Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3379554Z 
2025-09-07T09:06:09.3379655Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3379863Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3380002Z     return mod(**inputs)
2025-09-07T09:06:09.3380272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3380363Z     outputs = self.model(
2025-09-07T09:06:09.3380615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3380732Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3380980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3381052Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3381274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3381352Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3381609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward
2025-09-07T09:06:09.3381707Z     hidden_states, self_attn_weights = self.self_attn(
2025-09-07T09:06:09.3381962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3382049Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3382053Z 
2025-09-07T09:06:09.3382156Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3382361Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3382422Z     return mod(**inputs)
2025-09-07T09:06:09.3382678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3382739Z     outputs = self.model(
2025-09-07T09:06:09.3382982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3383055Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3383299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3383372Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3383591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3383664Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3383908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3384011Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3384258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward
2025-09-07T09:06:09.3384403Z     query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2)
2025-09-07T09:06:09.3384443Z 
2025-09-07T09:06:09.3384542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3384816Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3384873Z     return mod(**inputs)
2025-09-07T09:06:09.3385127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3385187Z     outputs = self.model(
2025-09-07T09:06:09.3385435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3385501Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3385744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3385810Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3386025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3386135Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3386381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3386477Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3386756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward
2025-09-07T09:06:09.3386828Z     key_states = self.k_proj(current_states)
2025-09-07T09:06:09.3386832Z 
2025-09-07T09:06:09.3386925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3387110Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3387171Z     return mod(**inputs)
2025-09-07T09:06:09.3387415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3387475Z     outputs = self.model(
2025-09-07T09:06:09.3387725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3387791Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3388039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3388103Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3388317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3388386Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3388627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3388722Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3388963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward
2025-09-07T09:06:09.3389045Z     value_states = self.v_proj(current_states)
2025-09-07T09:06:09.3389049Z 
2025-09-07T09:06:09.3389122Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3389192Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3389263Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3389329Z cudagraph partition due to non gpu ops
2025-09-07T09:06:09.3389426Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3389612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3389669Z     return mod(**inputs)
2025-09-07T09:06:09.3389916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3389973Z     outputs = self.model(
2025-09-07T09:06:09.3390219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3390319Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3390618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3390688Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3390911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3390984Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3391231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3391328Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3391573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3391666Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3391999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:06:09.3392127Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:09.3392131Z 
2025-09-07T09:06:09.3392229Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3392452Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3392512Z     return mod(**inputs)
2025-09-07T09:06:09.3392764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3392823Z     outputs = self.model(
2025-09-07T09:06:09.3393068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3393134Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3393380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3393451Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3393669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3393742Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3393988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3394088Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3394328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward
2025-09-07T09:06:09.3394410Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:06:09.3394703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:06:09.3394802Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:06:09.3394807Z 
2025-09-07T09:06:09.3394904Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3395092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3395150Z     return mod(**inputs)
2025-09-07T09:06:09.3395404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3395464Z     outputs = self.model(
2025-09-07T09:06:09.3395710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3395773Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3396020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3396089Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3396344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3396456Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3396704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward
2025-09-07T09:06:09.3396808Z     hidden_states, cross_attn_weights = self.encoder_attn(
2025-09-07T09:06:09.3397048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward
2025-09-07T09:06:09.3397119Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:06:09.3397123Z 
2025-09-07T09:06:09.3397218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3397403Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3397463Z     return mod(**inputs)
2025-09-07T09:06:09.3397747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3397808Z     outputs = self.model(
2025-09-07T09:06:09.3398058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3398121Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3398402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3398466Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3398688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3398755Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3398996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3399112Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3399117Z 
2025-09-07T09:06:09.3399208Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3399400Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3399457Z     return mod(**inputs)
2025-09-07T09:06:09.3399702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3399765Z     outputs = self.model(
2025-09-07T09:06:09.3400006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3400076Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3400317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3400378Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3400595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3400663Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3400906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward
2025-09-07T09:06:09.3401007Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:06:09.3401223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:09.3401285Z     return self.act(input)
2025-09-07T09:06:09.3401289Z 
2025-09-07T09:06:09.3401378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3401567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3401624Z     return mod(**inputs)
2025-09-07T09:06:09.3401877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward
2025-09-07T09:06:09.3401984Z     outputs = self.model(
2025-09-07T09:06:09.3402281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward
2025-09-07T09:06:09.3402351Z     decoder_outputs = self.decoder(
2025-09-07T09:06:09.3402596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward
2025-09-07T09:06:09.3402665Z     layer_outputs = decoder_layer(
2025-09-07T09:06:09.3402877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:09.3402945Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:09.3403361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward
2025-09-07T09:06:09.3403437Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:06:09.3403442Z 
2025-09-07T09:06:09.3403596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3403793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3403853Z     return mod(**inputs)
2025-09-07T09:06:09.3404098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1490, in forward
2025-09-07T09:06:09.3404223Z     lm_logits = self.lm_head(outputs[0])
2025-09-07T09:06:09.3404227Z 
2025-09-07T09:06:09.3404322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:09.3404504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:09.3404563Z     return mod(**inputs)
2025-09-07T09:06:09.3404806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1491, in forward
2025-09-07T09:06:09.3404926Z     lm_logits = lm_logits + self.final_logits_bias.to(lm_logits.device)
2025-09-07T09:06:09.3404934Z 
2025-09-07T09:06:19.8624172Z 
2025-09-07T09:06:20.0025637Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:06:20.1386153Z running benchmark:   3% 1/30 [00:00<00:04,  7.15it/s]
2025-09-07T09:06:20.2723719Z running benchmark:   7% 2/30 [00:00<00:03,  7.27it/s]
2025-09-07T09:06:20.4066381Z running benchmark:  10% 3/30 [00:00<00:03,  7.36it/s]
2025-09-07T09:06:20.5402290Z running benchmark:  13% 4/30 [00:00<00:03,  7.40it/s]
2025-09-07T09:06:20.6751431Z running benchmark:  17% 5/30 [00:00<00:03,  7.43it/s]
2025-09-07T09:06:20.8085669Z running benchmark:  20% 6/30 [00:00<00:03,  7.42it/s]
2025-09-07T09:06:20.9417275Z running benchmark:  23% 7/30 [00:00<00:03,  7.45it/s]
2025-09-07T09:06:21.0761498Z running benchmark:  27% 8/30 [00:01<00:02,  7.47it/s]
2025-09-07T09:06:21.2101370Z running benchmark:  30% 9/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:06:21.3440268Z running benchmark:  33% 10/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:06:21.4779277Z running benchmark:  37% 11/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:06:21.6122591Z running benchmark:  40% 12/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:06:21.7464310Z running benchmark:  43% 13/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:06:21.8804053Z running benchmark:  47% 14/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:06:22.0142791Z running benchmark:  50% 15/30 [00:02<00:02,  7.46it/s]
2025-09-07T09:06:22.1485363Z running benchmark:  53% 16/30 [00:02<00:01,  7.46it/s]
2025-09-07T09:06:22.2822079Z running benchmark:  57% 17/30 [00:02<00:01,  7.46it/s]
2025-09-07T09:06:22.4175655Z running benchmark:  60% 18/30 [00:02<00:01,  7.47it/s]
2025-09-07T09:06:22.5512676Z running benchmark:  63% 19/30 [00:02<00:01,  7.44it/s]
2025-09-07T09:06:22.6852467Z running benchmark:  67% 20/30 [00:02<00:01,  7.45it/s]
2025-09-07T09:06:22.8202777Z running benchmark:  70% 21/30 [00:02<00:01,  7.46it/s]
2025-09-07T09:06:22.9545809Z running benchmark:  73% 22/30 [00:02<00:01,  7.44it/s]
2025-09-07T09:06:23.0887675Z running benchmark:  77% 23/30 [00:03<00:00,  7.44it/s]
2025-09-07T09:06:23.2231841Z running benchmark:  80% 24/30 [00:03<00:00,  7.44it/s]
2025-09-07T09:06:23.3588301Z running benchmark:  83% 25/30 [00:03<00:00,  7.44it/s]
2025-09-07T09:06:23.4940802Z running benchmark:  87% 26/30 [00:03<00:00,  7.42it/s]
2025-09-07T09:06:23.6281217Z running benchmark:  90% 27/30 [00:03<00:00,  7.41it/s]
2025-09-07T09:06:23.7631418Z running benchmark:  93% 28/30 [00:03<00:00,  7.43it/s]
2025-09-07T09:06:23.8968212Z running benchmark:  97% 29/30 [00:03<00:00,  7.42it/s]
2025-09-07T09:06:23.8968792Z running benchmark: 100% 30/30 [00:04<00:00,  7.44it/s]
2025-09-07T09:06:23.8969323Z running benchmark: 100% 30/30 [00:04<00:00,  7.44it/s]
2025-09-07T09:06:24.0013744Z 1.191x
2025-09-07T09:06:24.0017956Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:06:26.4165396Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:06:26.4166370Z   import pynvml  # type: ignore[import]
2025-09-07T09:06:28.4413610Z 
2025-09-07T09:06:31.3225143Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:06:31.3225742Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:06:31.3408516Z cpu  eval  hf_Bert                            
2025-09-07T09:06:37.2261096Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2261400Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2261607Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2261814Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2262007Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2262208Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2262398Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2262588Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2262791Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2262989Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2263191Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2263381Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2263625Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2264011Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2264350Z     return mod(**inputs)
2025-09-07T09:06:37.2264750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2265159Z     outputs = self.bert(
2025-09-07T09:06:37.2265529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2265923Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2266304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2266723Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2267074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2267440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2267827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2268212Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2268598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2268973Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2269349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2269757Z     self_outputs = self.self(
2025-09-07T09:06:37.2270236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2270726Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2271115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2271656Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2271928Z 
2025-09-07T09:06:37.2272113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2272478Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2272805Z     return mod(**inputs)
2025-09-07T09:06:37.2273183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2273570Z     outputs = self.bert(
2025-09-07T09:06:37.2273995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2274389Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2274767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2275150Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2275529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2275983Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2276405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2276795Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2277175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2277554Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2277914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2278296Z     self_outputs = self.self(
2025-09-07T09:06:37.2278653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2279020Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2279372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2279748Z     self.key(current_states)
2025-09-07T09:06:37.2279862Z 
2025-09-07T09:06:37.2279964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2280321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2280641Z     return mod(**inputs)
2025-09-07T09:06:37.2280996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2281378Z     outputs = self.bert(
2025-09-07T09:06:37.2281736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2282122Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2282512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2282892Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2283233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2283596Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2283981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2284389Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2284814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2285238Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2285604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2285983Z     self_outputs = self.self(
2025-09-07T09:06:37.2286337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2286709Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2287065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2287454Z     self.value(current_states)
2025-09-07T09:06:37.2287565Z 
2025-09-07T09:06:37.2287653Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2287885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2288285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2288625Z     return mod(**inputs)
2025-09-07T09:06:37.2288985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2289354Z     outputs = self.bert(
2025-09-07T09:06:37.2289731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2290129Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2290533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2290920Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2291256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2291623Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2292004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2292390Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2292762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2293133Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2293497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2293872Z     self_outputs = self.self(
2025-09-07T09:06:37.2294230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2294586Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2294937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2295373Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2295557Z 
2025-09-07T09:06:37.2295685Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2296068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2296379Z     return mod(**inputs)
2025-09-07T09:06:37.2296734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2297129Z     outputs = self.bert(
2025-09-07T09:06:37.2297497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2297955Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2298338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2298717Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2299107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2299504Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2299874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2300258Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2300632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2301002Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2301360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2301799Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2302221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2302644Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2302782Z 
2025-09-07T09:06:37.2302887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2303429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2303761Z     return mod(**inputs)
2025-09-07T09:06:37.2304252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2304643Z     outputs = self.bert(
2025-09-07T09:06:37.2304993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2305368Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2305742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2306122Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2306470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2306824Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2307204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2307600Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2308009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2308405Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2308809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2309268Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2309693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2310085Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2310220Z 
2025-09-07T09:06:37.2310326Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2310684Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2311003Z     return mod(**inputs)
2025-09-07T09:06:37.2311366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2311748Z     outputs = self.bert(
2025-09-07T09:06:37.2312088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2312470Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2312838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2313283Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2313621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2314057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2314441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2314833Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2315239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2315636Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2316037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2316492Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2316983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2317409Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2317787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2318117Z     return self.act(input)
2025-09-07T09:06:37.2318231Z 
2025-09-07T09:06:37.2318364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2318720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2319036Z     return mod(**inputs)
2025-09-07T09:06:37.2319383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2319757Z     outputs = self.bert(
2025-09-07T09:06:37.2320098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2320482Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2320852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2321222Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2321564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2321923Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2322293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2322672Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2323073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2323469Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2323880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2324347Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2324779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2325165Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2325304Z 
2025-09-07T09:06:37.2325400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2325750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2326073Z     return mod(**inputs)
2025-09-07T09:06:37.2326413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2326786Z     outputs = self.bert(
2025-09-07T09:06:37.2327136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2327605Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2327980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2328365Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2328713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2329076Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2329461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2329842Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2330219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2330594Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2330995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2331380Z     self_outputs = self.self(
2025-09-07T09:06:37.2331735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2332104Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2332497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2333025Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2333292Z 
2025-09-07T09:06:37.2333393Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2333750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2334070Z     return mod(**inputs)
2025-09-07T09:06:37.2334434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2334807Z     outputs = self.bert(
2025-09-07T09:06:37.2335145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2335538Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2335913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2336295Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2336661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2337014Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2337388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2337852Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2338235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2338605Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2338959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2339329Z     self_outputs = self.self(
2025-09-07T09:06:37.2339691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2340063Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2340407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2340789Z     self.key(current_states)
2025-09-07T09:06:37.2340903Z 
2025-09-07T09:06:37.2341024Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2341431Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2341784Z     return mod(**inputs)
2025-09-07T09:06:37.2342141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2342522Z     outputs = self.bert(
2025-09-07T09:06:37.2342866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2343246Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2343608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2343984Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2344321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2344682Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2345087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2345481Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2345854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2346220Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2346609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2346987Z     self_outputs = self.self(
2025-09-07T09:06:37.2347340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2347701Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2348055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2348429Z     self.value(current_states)
2025-09-07T09:06:37.2348541Z 
2025-09-07T09:06:37.2348619Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2348852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2349205Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2349524Z     return mod(**inputs)
2025-09-07T09:06:37.2349878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2350252Z     outputs = self.bert(
2025-09-07T09:06:37.2350595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2350975Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2351344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2351713Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2352052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2352406Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2352781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2353153Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2353528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2353905Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2354320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2354699Z     self_outputs = self.self(
2025-09-07T09:06:37.2355049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2355463Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2355855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2356291Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2356475Z 
2025-09-07T09:06:37.2356580Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2356930Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2357256Z     return mod(**inputs)
2025-09-07T09:06:37.2357607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2357979Z     outputs = self.bert(
2025-09-07T09:06:37.2358316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2358700Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2359098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2359485Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2359823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2360171Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2360580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2360970Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2361340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2361707Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2362062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2362496Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2362919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2363303Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2363438Z 
2025-09-07T09:06:37.2363538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2363886Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2364205Z     return mod(**inputs)
2025-09-07T09:06:37.2364560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2364933Z     outputs = self.bert(
2025-09-07T09:06:37.2365273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2365648Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2366011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2366388Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2366720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2367081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2367452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2367849Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2368251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2368643Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2369064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2370625Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2371079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2371480Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2371620Z 
2025-09-07T09:06:37.2371726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2372091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2372420Z     return mod(**inputs)
2025-09-07T09:06:37.2372783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2373157Z     outputs = self.bert(
2025-09-07T09:06:37.2373505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2373936Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2374317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2374697Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2375040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2375436Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2375819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2376211Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2376610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2377002Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2377415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2377959Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2378382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2378799Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2379180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2379521Z     return self.act(input)
2025-09-07T09:06:37.2379629Z 
2025-09-07T09:06:37.2379734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2380092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2380404Z     return mod(**inputs)
2025-09-07T09:06:37.2380761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2381136Z     outputs = self.bert(
2025-09-07T09:06:37.2381484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2393118Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2393611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2394030Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2394399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2394769Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2395166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2395571Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2396046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2396502Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2396921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2397390Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2397839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2398237Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2398376Z 
2025-09-07T09:06:37.2398492Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2398871Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2399193Z     return mod(**inputs)
2025-09-07T09:06:37.2399606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2400000Z     outputs = self.bert(
2025-09-07T09:06:37.2400350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2400737Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2401148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2401537Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2401884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2402245Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2402620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2403009Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2403553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2403934Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2404299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2404673Z     self_outputs = self.self(
2025-09-07T09:06:37.2405035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2405406Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2405769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2406283Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2406556Z 
2025-09-07T09:06:37.2406664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2407031Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2407355Z     return mod(**inputs)
2025-09-07T09:06:37.2407714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2408085Z     outputs = self.bert(
2025-09-07T09:06:37.2408431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2408815Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2409186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2409562Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2409898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2410358Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2410737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2411187Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2411563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2411934Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2412300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2412679Z     self_outputs = self.self(
2025-09-07T09:06:37.2413035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2413392Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2413751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2414171Z     self.key(current_states)
2025-09-07T09:06:37.2414283Z 
2025-09-07T09:06:37.2414389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2414748Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2415062Z     return mod(**inputs)
2025-09-07T09:06:37.2415463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2415849Z     outputs = self.bert(
2025-09-07T09:06:37.2416196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2416570Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2416943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2417322Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2417739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2418102Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2418474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2418857Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2419230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2419596Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2419949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2420323Z     self_outputs = self.self(
2025-09-07T09:06:37.2420668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2421036Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2421388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2421757Z     self.value(current_states)
2025-09-07T09:06:37.2421875Z 
2025-09-07T09:06:37.2421951Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2422184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2422538Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2422846Z     return mod(**inputs)
2025-09-07T09:06:37.2423200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2423572Z     outputs = self.bert(
2025-09-07T09:06:37.2423912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2424347Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2424710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2425125Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2425460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2425815Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2426181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2426565Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2426932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2427296Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2427650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2428064Z     self_outputs = self.self(
2025-09-07T09:06:37.2428419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2428784Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2429168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2429633Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2429814Z 
2025-09-07T09:06:37.2429911Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2430256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2430582Z     return mod(**inputs)
2025-09-07T09:06:37.2430954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2431328Z     outputs = self.bert(
2025-09-07T09:06:37.2431675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2432059Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2432423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2432800Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2433134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2433495Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2433871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2434255Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2434628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2434994Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2435355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2435784Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2436207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2436586Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2436727Z 
2025-09-07T09:06:37.2436824Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2437179Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2437494Z     return mod(**inputs)
2025-09-07T09:06:37.2437836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2438251Z     outputs = self.bert(
2025-09-07T09:06:37.2438663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2439041Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2439408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2439782Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2440112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2440464Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2440836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2441221Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2441664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2442070Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2442489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2442944Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2443403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2443791Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2443935Z 
2025-09-07T09:06:37.2444033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2444388Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2444717Z     return mod(**inputs)
2025-09-07T09:06:37.2445089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2445466Z     outputs = self.bert(
2025-09-07T09:06:37.2445812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2446188Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2446563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2446943Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2447286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2447649Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2448028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2448415Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2448815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2449206Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2449606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2450057Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2450475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2450885Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2451253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2451584Z     return self.act(input)
2025-09-07T09:06:37.2451702Z 
2025-09-07T09:06:37.2451803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2452204Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2452552Z     return mod(**inputs)
2025-09-07T09:06:37.2452898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2453266Z     outputs = self.bert(
2025-09-07T09:06:37.2453601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2453970Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2454329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2454692Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2455023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2455369Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2455767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2456151Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2456543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2456931Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2457359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2457876Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2458294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2458673Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2458806Z 
2025-09-07T09:06:37.2458897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2459238Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2459542Z     return mod(**inputs)
2025-09-07T09:06:37.2459880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2460245Z     outputs = self.bert(
2025-09-07T09:06:37.2460583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2460952Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2461304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2461672Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2462001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2462344Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2462707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2463083Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2463457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2463818Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2464166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2464525Z     self_outputs = self.self(
2025-09-07T09:06:37.2464865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2465219Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2465562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2466130Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2466427Z 
2025-09-07T09:06:37.2466522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2466864Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2467170Z     return mod(**inputs)
2025-09-07T09:06:37.2467512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2467875Z     outputs = self.bert(
2025-09-07T09:06:37.2468210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2468580Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2468931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2469354Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2469682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2470033Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2470403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2470809Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2471170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2471522Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2471863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2472225Z     self_outputs = self.self(
2025-09-07T09:06:37.2472561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2472917Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2473258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2473620Z     self.key(current_states)
2025-09-07T09:06:37.2473725Z 
2025-09-07T09:06:37.2473820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2474157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2474457Z     return mod(**inputs)
2025-09-07T09:06:37.2474792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2475155Z     outputs = self.bert(
2025-09-07T09:06:37.2475488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2475853Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2476214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2476583Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2476917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2477260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2477615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2477982Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2478343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2478701Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2479045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2479450Z     self_outputs = self.self(
2025-09-07T09:06:37.2479817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2480170Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2480513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2480880Z     self.value(current_states)
2025-09-07T09:06:37.2480993Z 
2025-09-07T09:06:37.2481065Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2481282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2481615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2481921Z     return mod(**inputs)
2025-09-07T09:06:37.2482258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2482660Z     outputs = self.bert(
2025-09-07T09:06:37.2482999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2483370Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2483724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2484126Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2484455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2484814Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2485173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2485539Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2485899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2486254Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2486594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2486954Z     self_outputs = self.self(
2025-09-07T09:06:37.2487294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2487652Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2487997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2488415Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2488592Z 
2025-09-07T09:06:37.2488816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2489148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2489454Z     return mod(**inputs)
2025-09-07T09:06:37.2489789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2490151Z     outputs = self.bert(
2025-09-07T09:06:37.2490482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2490856Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2491207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2491574Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2491894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2492240Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2492611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2493104Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2493481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2493842Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2494207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2494643Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2495062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2495438Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2495570Z 
2025-09-07T09:06:37.2495666Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2496010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2496358Z     return mod(**inputs)
2025-09-07T09:06:37.2496707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2497071Z     outputs = self.bert(
2025-09-07T09:06:37.2497413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2497882Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2498261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2498641Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2498986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2499342Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2499717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2500104Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2500497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2500881Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2501285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2501732Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2502143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2502519Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2502647Z 
2025-09-07T09:06:37.2502740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2503214Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2503528Z     return mod(**inputs)
2025-09-07T09:06:37.2503866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2504230Z     outputs = self.bert(
2025-09-07T09:06:37.2504569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2504942Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2505300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2505669Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2505998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2506346Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2506792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2507239Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2507629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2508014Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2508416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2508860Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2509276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2509683Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2510057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2510441Z     return self.act(input)
2025-09-07T09:06:37.2510546Z 
2025-09-07T09:06:37.2510651Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2510993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2511307Z     return mod(**inputs)
2025-09-07T09:06:37.2511702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2512081Z     outputs = self.bert(
2025-09-07T09:06:37.2512425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2512796Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2513165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2513535Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2513882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2514238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2514607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2514990Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2515387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2515778Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2516182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2516639Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2517079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2517473Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2517605Z 
2025-09-07T09:06:37.2517709Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2518054Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2518370Z     return mod(**inputs)
2025-09-07T09:06:37.2518717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2519093Z     outputs = self.bert(
2025-09-07T09:06:37.2519445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2519819Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2520188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2520613Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2520946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2521402Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2521771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2522154Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2522526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2522893Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2523246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2523620Z     self_outputs = self.self(
2025-09-07T09:06:37.2523969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2524369Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2524728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2525243Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2525509Z 
2025-09-07T09:06:37.2525638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2525986Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2526300Z     return mod(**inputs)
2025-09-07T09:06:37.2526646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2527011Z     outputs = self.bert(
2025-09-07T09:06:37.2527353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2527736Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2528104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2528472Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2528810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2529166Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2529538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2529915Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2530278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2530643Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2530993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2531375Z     self_outputs = self.self(
2025-09-07T09:06:37.2531713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2532068Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2532410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2532773Z     self.key(current_states)
2025-09-07T09:06:37.2532883Z 
2025-09-07T09:06:37.2532982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2533313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2533620Z     return mod(**inputs)
2025-09-07T09:06:37.2533957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2534367Z     outputs = self.bert(
2025-09-07T09:06:37.2534696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2535103Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2535460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2535828Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2536149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2536491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2536860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2537232Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2537713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2538123Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2538517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2538945Z     self_outputs = self.self(
2025-09-07T09:06:37.2539336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2539709Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2540070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2540440Z     self.value(current_states)
2025-09-07T09:06:37.2540554Z 
2025-09-07T09:06:37.2540628Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2540856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2541204Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2541515Z     return mod(**inputs)
2025-09-07T09:06:37.2541872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2542242Z     outputs = self.bert(
2025-09-07T09:06:37.2542599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2542979Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2543352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2543723Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2544067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2544428Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2544800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2545192Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2545563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2545946Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2546301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2546664Z     self_outputs = self.self(
2025-09-07T09:06:37.2547015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2547370Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2547718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2548135Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2548396Z 
2025-09-07T09:06:37.2548495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2548884Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2549217Z     return mod(**inputs)
2025-09-07T09:06:37.2549641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2550035Z     outputs = self.bert(
2025-09-07T09:06:37.2550395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2550797Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2551158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2551518Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2551902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2552264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2552632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2553021Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2553418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2553800Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2554161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2554606Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2555025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2555422Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2555561Z 
2025-09-07T09:06:37.2555659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2556006Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2556333Z     return mod(**inputs)
2025-09-07T09:06:37.2556682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2557061Z     outputs = self.bert(
2025-09-07T09:06:37.2557400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2557782Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2558144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2558510Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2558854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2559208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2559574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2559953Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2560353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2560740Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2561168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2561625Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2562035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2562706Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2562871Z 
2025-09-07T09:06:37.2562967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2563324Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2563634Z     return mod(**inputs)
2025-09-07T09:06:37.2563973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2564364Z     outputs = self.bert(
2025-09-07T09:06:37.2564701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2565079Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2565461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2565824Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2566222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2566582Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2566957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2567333Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2567758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2568162Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2568584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2569043Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2569456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2569892Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2570279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2570620Z     return self.act(input)
2025-09-07T09:06:37.2570722Z 
2025-09-07T09:06:37.2570823Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2571166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2571485Z     return mod(**inputs)
2025-09-07T09:06:37.2571830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2572215Z     outputs = self.bert(
2025-09-07T09:06:37.2572550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2572945Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2573320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2573695Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2574026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2574372Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2574758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2575156Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2575564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2575944Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2576364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2576880Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2577351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2577865Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2577994Z 
2025-09-07T09:06:37.2578090Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2578442Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2578751Z     return mod(**inputs)
2025-09-07T09:06:37.2579087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2579453Z     outputs = self.bert(
2025-09-07T09:06:37.2579784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2580191Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2580552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2580916Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2581240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2581619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2582000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2582387Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2582764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2583129Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2583493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2583868Z     self_outputs = self.self(
2025-09-07T09:06:37.2584223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2584585Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2584938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2585456Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2585723Z 
2025-09-07T09:06:37.2585818Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2586170Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2586479Z     return mod(**inputs)
2025-09-07T09:06:37.2586832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2587212Z     outputs = self.bert(
2025-09-07T09:06:37.2587554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2587931Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2588290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2588662Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2588994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2589343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2589714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2590092Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2590511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2590908Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2591261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2591631Z     self_outputs = self.self(
2025-09-07T09:06:37.2591982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2592347Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2592696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2593068Z     self.key(current_states)
2025-09-07T09:06:37.2593176Z 
2025-09-07T09:06:37.2593272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2593618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2593965Z     return mod(**inputs)
2025-09-07T09:06:37.2594320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2594687Z     outputs = self.bert(
2025-09-07T09:06:37.2595035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2595467Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2595837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2596212Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2596540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2596895Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2597268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2597655Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2598030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2598390Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2598755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2599130Z     self_outputs = self.self(
2025-09-07T09:06:37.2599483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2599839Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2600199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2600579Z     self.value(current_states)
2025-09-07T09:06:37.2600690Z 
2025-09-07T09:06:37.2600774Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2601003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2601347Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2601662Z     return mod(**inputs)
2025-09-07T09:06:37.2602010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2602384Z     outputs = self.bert(
2025-09-07T09:06:37.2602724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2603255Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2603643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2604036Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2604383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2604870Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2605248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2605641Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2606021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2606386Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2606747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2607124Z     self_outputs = self.self(
2025-09-07T09:06:37.2607480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2607844Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2608238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2608683Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2608874Z 
2025-09-07T09:06:37.2608976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2609386Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2609708Z     return mod(**inputs)
2025-09-07T09:06:37.2610057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2610432Z     outputs = self.bert(
2025-09-07T09:06:37.2610788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2611172Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2611540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2611919Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2612260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2612619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2612999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2613375Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2613747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2614119Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2614476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2614898Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2615329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2615717Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2615860Z 
2025-09-07T09:06:37.2615960Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2616313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2616623Z     return mod(**inputs)
2025-09-07T09:06:37.2616981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2617358Z     outputs = self.bert(
2025-09-07T09:06:37.2617751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2618131Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2618542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2618965Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2619309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2619669Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2620039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2620429Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2620826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2621217Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2621623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2622111Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2622538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2622926Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2623059Z 
2025-09-07T09:06:37.2623162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2623548Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2623862Z     return mod(**inputs)
2025-09-07T09:06:37.2624215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2624585Z     outputs = self.bert(
2025-09-07T09:06:37.2624927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2625300Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2625672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2626044Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2626386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2626743Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2627113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2627499Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2627897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2628292Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2628695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2629147Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2629562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2629977Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2630354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2630681Z     return self.act(input)
2025-09-07T09:06:37.2630792Z 
2025-09-07T09:06:37.2630885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2631234Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2631551Z     return mod(**inputs)
2025-09-07T09:06:37.2631893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2632306Z     outputs = self.bert(
2025-09-07T09:06:37.2632647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2633079Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2633444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2633819Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2634154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2634511Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2634892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2635274Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2635698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2636087Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2636482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2636933Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2637398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2637779Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2637909Z 
2025-09-07T09:06:37.2637999Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2638337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2638649Z     return mod(**inputs)
2025-09-07T09:06:37.2638993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2639362Z     outputs = self.bert(
2025-09-07T09:06:37.2639706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2640081Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2640443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2640812Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2641149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2641500Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2641874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2642254Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2642624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2642995Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2643355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2643727Z     self_outputs = self.self(
2025-09-07T09:06:37.2644072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2644432Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2644786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2645304Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2645566Z 
2025-09-07T09:06:37.2645665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2646073Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2646428Z     return mod(**inputs)
2025-09-07T09:06:37.2646778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2647153Z     outputs = self.bert(
2025-09-07T09:06:37.2647495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2647865Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2648225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2648598Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2648933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2649289Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2649733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2650125Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2650489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2650551Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2650830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2650891Z     self_outputs = self.self(
2025-09-07T09:06:37.2651127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2651186Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2651424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2651488Z     self.key(current_states)
2025-09-07T09:06:37.2651493Z 
2025-09-07T09:06:37.2651585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2651772Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2651827Z     return mod(**inputs)
2025-09-07T09:06:37.2652072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2652130Z     outputs = self.bert(
2025-09-07T09:06:37.2652371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2652436Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2652675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2652734Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2652950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2653018Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2653257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2653325Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2653560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2653620Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2653857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2653921Z     self_outputs = self.self(
2025-09-07T09:06:37.2654156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2654218Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2654507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2654619Z     self.value(current_states)
2025-09-07T09:06:37.2654622Z 
2025-09-07T09:06:37.2654699Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2654795Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2654990Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2655045Z     return mod(**inputs)
2025-09-07T09:06:37.2655294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2655353Z     outputs = self.bert(
2025-09-07T09:06:37.2655596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2655666Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2655944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2656009Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2656235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2656308Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2656589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2656661Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2656905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2656969Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2657208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2657278Z     self_outputs = self.self(
2025-09-07T09:06:37.2657512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2657610Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2657851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2657980Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2657984Z 
2025-09-07T09:06:37.2658086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2658280Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2658343Z     return mod(**inputs)
2025-09-07T09:06:37.2658589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2658650Z     outputs = self.bert(
2025-09-07T09:06:37.2658906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2658975Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2659223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2659286Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2659512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2659581Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2659822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2667490Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2667818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2667987Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2668288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2668459Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2668732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2668822Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2668827Z 
2025-09-07T09:06:37.2668931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2669143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2669206Z     return mod(**inputs)
2025-09-07T09:06:37.2669474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2669534Z     outputs = self.bert(
2025-09-07T09:06:37.2669823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2669898Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2670143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2670210Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2670467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2670543Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2670799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2670879Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2671149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2671222Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2671510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2671627Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2671872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2671946Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2671950Z 
2025-09-07T09:06:37.2672046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2672247Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2672303Z     return mod(**inputs)
2025-09-07T09:06:37.2672552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2672612Z     outputs = self.bert(
2025-09-07T09:06:37.2672857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2672930Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2673171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2673235Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2673455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2673525Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2673769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2673842Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2674100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2674211Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2674521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2674633Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2674877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2674982Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2675189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2675249Z     return self.act(input)
2025-09-07T09:06:37.2675253Z 
2025-09-07T09:06:37.2675350Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2675542Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2675636Z     return mod(**inputs)
2025-09-07T09:06:37.2675886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2675945Z     outputs = self.bert(
2025-09-07T09:06:37.2676190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2676288Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2676532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2676594Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2676812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2676878Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2677118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2677193Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2677447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2677520Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2677796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2677923Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2678167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2678242Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2678245Z 
2025-09-07T09:06:37.2678341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2678531Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2678594Z     return mod(**inputs)
2025-09-07T09:06:37.2678836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2678892Z     outputs = self.bert(
2025-09-07T09:06:37.2679138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2679201Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2679448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2679508Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2679723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2679794Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2680073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2680208Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2680450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2680516Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2680765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2680829Z     self_outputs = self.self(
2025-09-07T09:06:37.2681068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2681126Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2681373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2681616Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2681623Z 
2025-09-07T09:06:37.2681715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2681908Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2681967Z     return mod(**inputs)
2025-09-07T09:06:37.2682253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2682309Z     outputs = self.bert(
2025-09-07T09:06:37.2682556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2682625Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2682864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2682931Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2683148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2683223Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2683463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2683531Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2683777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2683839Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2684084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2684142Z     self_outputs = self.self(
2025-09-07T09:06:37.2684378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2684444Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2684683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2684755Z     self.key(current_states)
2025-09-07T09:06:37.2684758Z 
2025-09-07T09:06:37.2684846Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2685036Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2685099Z     return mod(**inputs)
2025-09-07T09:06:37.2685343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2685406Z     outputs = self.bert(
2025-09-07T09:06:37.2685647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2685714Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2685997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2686091Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2686311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2686377Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2686631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2686701Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2686933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2686996Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2687235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2687298Z     self_outputs = self.self(
2025-09-07T09:06:37.2687562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2687625Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2687871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2687932Z     self.value(current_states)
2025-09-07T09:06:37.2687968Z 
2025-09-07T09:06:37.2688043Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2688129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2688313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2688369Z     return mod(**inputs)
2025-09-07T09:06:37.2688616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2688674Z     outputs = self.bert(
2025-09-07T09:06:37.2688918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2688982Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2689221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2689281Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2689500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2689563Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2689805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2689872Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2690106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2690171Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2690411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2690474Z     self_outputs = self.self(
2025-09-07T09:06:37.2690707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2690767Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2691009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2691132Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2691135Z 
2025-09-07T09:06:37.2691227Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2691409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2691468Z     return mod(**inputs)
2025-09-07T09:06:37.2691750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2691838Z     outputs = self.bert(
2025-09-07T09:06:37.2692088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2692151Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2692397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2692458Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2692671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2692739Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2692977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2693082Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2693319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2693386Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2693628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2694006Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2694255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2694331Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2694334Z 
2025-09-07T09:06:37.2694431Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2694616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2694676Z     return mod(**inputs)
2025-09-07T09:06:37.2694926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2694983Z     outputs = self.bert(
2025-09-07T09:06:37.2695227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2695291Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2695532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2695600Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2695811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2695882Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2696120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2696200Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2696461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2696528Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2696810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2696920Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2697165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2697235Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2697238Z 
2025-09-07T09:06:37.2697326Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2697514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2697751Z     return mod(**inputs)
2025-09-07T09:06:37.2698040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2698098Z     outputs = self.bert(
2025-09-07T09:06:37.2698357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2698420Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2698662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2698726Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2698940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2699008Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2699284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2699357Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2699619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2699684Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2699996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2700102Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2700346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2700449Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2700655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2700719Z     return self.act(input)
2025-09-07T09:06:37.2700724Z 
2025-09-07T09:06:37.2700813Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2701008Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2701063Z     return mod(**inputs)
2025-09-07T09:06:37.2701308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2701366Z     outputs = self.bert(
2025-09-07T09:06:37.2701607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2701669Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2701907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2701965Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2702185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2702254Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2702496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2702564Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2702819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2702882Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2703313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2703440Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2703681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2703850Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2703901Z 
2025-09-07T09:06:37.2703990Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2704172Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2704229Z     return mod(**inputs)
2025-09-07T09:06:37.2704474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2704532Z     outputs = self.bert(
2025-09-07T09:06:37.2704774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2704834Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2705074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2705133Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2705393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2705460Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2705703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2705772Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2706060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2706122Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2706363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2706424Z     self_outputs = self.self(
2025-09-07T09:06:37.2706657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2706716Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2706960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2707163Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2707167Z 
2025-09-07T09:06:37.2707257Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2707440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2707500Z     return mod(**inputs)
2025-09-07T09:06:37.2707744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2707801Z     outputs = self.bert(
2025-09-07T09:06:37.2708047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2708112Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2708356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2708418Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2708631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2708701Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2708944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2709018Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2709253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2709313Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2709552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2709650Z     self_outputs = self.self(
2025-09-07T09:06:37.2709924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2709984Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2710227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2710296Z     self.key(current_states)
2025-09-07T09:06:37.2710299Z 
2025-09-07T09:06:37.2710385Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2710568Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2710620Z     return mod(**inputs)
2025-09-07T09:06:37.2710864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2710919Z     outputs = self.bert(
2025-09-07T09:06:37.2711196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2711264Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2711504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2711571Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2711813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2711878Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2712125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2712198Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2712434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2712497Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2712738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2712807Z     self_outputs = self.self(
2025-09-07T09:06:37.2713042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2713108Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2713346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2713413Z     self.value(current_states)
2025-09-07T09:06:37.2713417Z 
2025-09-07T09:06:37.2713488Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2713577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2713760Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2713814Z     return mod(**inputs)
2025-09-07T09:06:37.2714059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2714115Z     outputs = self.bert(
2025-09-07T09:06:37.2714357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2714424Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2714662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2714723Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2714932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2714996Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2715235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2715341Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2715609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2715667Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2715912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2715977Z     self_outputs = self.self(
2025-09-07T09:06:37.2716216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2716277Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2716516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2716642Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2716646Z 
2025-09-07T09:06:37.2716773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2716968Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2717029Z     return mod(**inputs)
2025-09-07T09:06:37.2717274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2717331Z     outputs = self.bert(
2025-09-07T09:06:37.2717607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2717671Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2717913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2717974Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2718192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2718264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2718505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2718576Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2718809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2718871Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2719110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2719229Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2719467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2719539Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2719544Z 
2025-09-07T09:06:37.2719636Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2719821Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2719878Z     return mod(**inputs)
2025-09-07T09:06:37.2720120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2720178Z     outputs = self.bert(
2025-09-07T09:06:37.2720422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2720482Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2720723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2720782Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2720995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2721095Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2721367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2721441Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2721703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2721771Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2722049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2722159Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2722402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2722471Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2722477Z 
2025-09-07T09:06:37.2722602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2722791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2722848Z     return mod(**inputs)
2025-09-07T09:06:37.2723093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2723196Z     outputs = self.bert(
2025-09-07T09:06:37.2723446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2723507Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2723748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2723807Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2724020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2724094Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2724332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2724405Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2724662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2724730Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2725003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2725107Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2725348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2725456Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2725669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2725729Z     return self.act(input)
2025-09-07T09:06:37.2725732Z 
2025-09-07T09:06:37.2725823Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2726013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2726070Z     return mod(**inputs)
2025-09-07T09:06:37.2726315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2726371Z     outputs = self.bert(
2025-09-07T09:06:37.2726612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2726678Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2726918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2727047Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2727261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2727326Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2727572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2727642Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2727902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2727965Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2728238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2728396Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2728644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2728722Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2728725Z 
2025-09-07T09:06:37.2728812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2729029Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2729086Z     return mod(**inputs)
2025-09-07T09:06:37.2729331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2729391Z     outputs = self.bert(
2025-09-07T09:06:37.2729634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2729697Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2729940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2730006Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2730217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2730282Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2730524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2730594Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2730840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2730899Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2731137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2731203Z     self_outputs = self.self(
2025-09-07T09:06:37.2731440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2731501Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2731741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2731943Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2731951Z 
2025-09-07T09:06:37.2732039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2732224Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2732283Z     return mod(**inputs)
2025-09-07T09:06:37.2732528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2732625Z     outputs = self.bert(
2025-09-07T09:06:37.2732869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2732966Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2733210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2733270Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2733485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2733549Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2733788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2733861Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2734129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2734193Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2734436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2734503Z     self_outputs = self.self(
2025-09-07T09:06:37.2734767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2734828Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2735069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2735130Z     self.key(current_states)
2025-09-07T09:06:37.2735133Z 
2025-09-07T09:06:37.2735224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2735408Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2735463Z     return mod(**inputs)
2025-09-07T09:06:37.2735709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2735764Z     outputs = self.bert(
2025-09-07T09:06:37.2736006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2736068Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2736308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2736374Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2736583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2736651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2736890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2736966Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2737201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2737260Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2737499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2737594Z     self_outputs = self.self(
2025-09-07T09:06:37.2737834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2737891Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2738130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2738196Z     self.value(current_states)
2025-09-07T09:06:37.2738199Z 
2025-09-07T09:06:37.2738269Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2738400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2738616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2738671Z     return mod(**inputs)
2025-09-07T09:06:37.2738918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2738974Z     outputs = self.bert(
2025-09-07T09:06:37.2739222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2739282Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2739521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2739585Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2739796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2739900Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2740144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2740218Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2740486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2740546Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2740792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2740850Z     self_outputs = self.self(
2025-09-07T09:06:37.2741084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2741139Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2741380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2741509Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2741513Z 
2025-09-07T09:06:37.2741601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2741786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2741841Z     return mod(**inputs)
2025-09-07T09:06:37.2742084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2742144Z     outputs = self.bert(
2025-09-07T09:06:37.2742389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2742452Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2742691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2742758Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2742968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2743032Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2743276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2743345Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2743578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2743639Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2743878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2743998Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2744285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2744392Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2744396Z 
2025-09-07T09:06:37.2744481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2744666Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2744723Z     return mod(**inputs)
2025-09-07T09:06:37.2744963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2745020Z     outputs = self.bert(
2025-09-07T09:06:37.2745262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2745325Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2745613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2745675Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2745894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2745960Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2746246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2746318Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2746578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2746645Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2746919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2747032Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2747272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2747348Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2747351Z 
2025-09-07T09:06:37.2747438Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2747624Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2747682Z     return mod(**inputs)
2025-09-07T09:06:37.2747927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2747986Z     outputs = self.bert(
2025-09-07T09:06:37.2748230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2748290Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2748532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2748595Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2748809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2748875Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2749117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2749191Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2749445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2749510Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2749779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2749938Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2750212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2750310Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2750520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2750582Z     return self.act(input)
2025-09-07T09:06:37.2750586Z 
2025-09-07T09:06:37.2750676Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2750856Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2750911Z     return mod(**inputs)
2025-09-07T09:06:37.2751159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2751215Z     outputs = self.bert(
2025-09-07T09:06:37.2751505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2751568Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2751814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2751873Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2752117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2752188Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2752430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2752503Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2752754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2752820Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2753094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2753215Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2753463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2753531Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2753534Z 
2025-09-07T09:06:37.2753624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2753805Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2753859Z     return mod(**inputs)
2025-09-07T09:06:37.2754107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2754163Z     outputs = self.bert(
2025-09-07T09:06:37.2754410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2754472Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2754709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2754771Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2754983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2755051Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2755288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2755356Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2755593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2755693Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2755992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2756053Z     self_outputs = self.self(
2025-09-07T09:06:37.2756292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2756351Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2756591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2756791Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2756795Z 
2025-09-07T09:06:37.2756883Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2757069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2757157Z     return mod(**inputs)
2025-09-07T09:06:37.2757407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2757466Z     outputs = self.bert(
2025-09-07T09:06:37.2757709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2757809Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2758050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2758111Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2758325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2758391Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2758632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2758701Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2758939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2758996Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2759237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2759300Z     self_outputs = self.self(
2025-09-07T09:06:37.2759533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2759592Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2759829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2759889Z     self.key(current_states)
2025-09-07T09:06:37.2759894Z 
2025-09-07T09:06:37.2759987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2760170Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2760229Z     return mod(**inputs)
2025-09-07T09:06:37.2760470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2760526Z     outputs = self.bert(
2025-09-07T09:06:37.2760770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2760831Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2761070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2761128Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2761340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2761462Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2761735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2761805Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2762039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2762101Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2762340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2762400Z     self_outputs = self.self(
2025-09-07T09:06:37.2762636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2762693Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2762968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2763034Z     self.value(current_states)
2025-09-07T09:06:37.2763037Z 
2025-09-07T09:06:37.2763107Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2763198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2763378Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2763468Z     return mod(**inputs)
2025-09-07T09:06:37.2763716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2763777Z     outputs = self.bert(
2025-09-07T09:06:37.2764020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2764081Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2764325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2764386Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2764600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2764663Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2764903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2764976Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2765209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2765269Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2765508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2765565Z     self_outputs = self.self(
2025-09-07T09:06:37.2765801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2765861Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2766105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2766221Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2766225Z 
2025-09-07T09:06:37.2766317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2766501Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2766554Z     return mod(**inputs)
2025-09-07T09:06:37.2766804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2766858Z     outputs = self.bert(
2025-09-07T09:06:37.2767105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2767200Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2767493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2767568Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2767784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2767856Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2768096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2768164Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2768404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2768462Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2768904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2769022Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2769270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2769343Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2769382Z 
2025-09-07T09:06:37.2769476Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2769666Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2769722Z     return mod(**inputs)
2025-09-07T09:06:37.2769972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2770029Z     outputs = self.bert(
2025-09-07T09:06:37.2770276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2770345Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2770587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2770653Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2770864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2770934Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2771174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2771246Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2771509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2771578Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2771864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2771975Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2772216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2772291Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2772295Z 
2025-09-07T09:06:37.2772382Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2772571Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2772628Z     return mod(**inputs)
2025-09-07T09:06:37.2772869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2772934Z     outputs = self.bert(
2025-09-07T09:06:37.2773217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2773322Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2773563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2773629Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2773844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2773913Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2774156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2774228Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2774489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2774588Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2774866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2774976Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2775219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2775357Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2775567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2775635Z     return self.act(input)
2025-09-07T09:06:37.2775639Z 
2025-09-07T09:06:37.2775727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2775912Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2775974Z     return mod(**inputs)
2025-09-07T09:06:37.2776221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2776286Z     outputs = self.bert(
2025-09-07T09:06:37.2776531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2776594Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2776842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2776902Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2777117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2777182Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2777424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2777503Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2777833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2777906Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2778175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2778304Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2778544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2778615Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2778618Z 
2025-09-07T09:06:37.2778716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2778899Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2779002Z     return mod(**inputs)
2025-09-07T09:06:37.2779247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2779341Z     outputs = self.bert(
2025-09-07T09:06:37.2779592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2779661Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2779906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2779966Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2780183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2780250Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2780489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2780603Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2780846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2780912Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2781185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2781248Z     self_outputs = self.self(
2025-09-07T09:06:37.2781492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2781548Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2781792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:06:37.2781993Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:06:37.2782000Z 
2025-09-07T09:06:37.2782095Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2782279Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2782337Z     return mod(**inputs)
2025-09-07T09:06:37.2782590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2782648Z     outputs = self.bert(
2025-09-07T09:06:37.2782897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2782960Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2783199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2783266Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2783479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2783556Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2783795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2783867Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2784110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2784170Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2784415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2784475Z     self_outputs = self.self(
2025-09-07T09:06:37.2784707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2784772Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2785047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:06:37.2785148Z     self.key(current_states)
2025-09-07T09:06:37.2785151Z 
2025-09-07T09:06:37.2785239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2785427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2785484Z     return mod(**inputs)
2025-09-07T09:06:37.2785729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2785791Z     outputs = self.bert(
2025-09-07T09:06:37.2786033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2786099Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2786371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2786436Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2786657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2786722Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2787066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2787135Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2787375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2787437Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2787680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2787745Z     self_outputs = self.self(
2025-09-07T09:06:37.2787979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2788048Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2788292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:06:37.2788357Z     self.value(current_states)
2025-09-07T09:06:37.2788360Z 
2025-09-07T09:06:37.2788438Z cudagraph partition due to non gpu ops
2025-09-07T09:06:37.2788533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2788727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2788784Z     return mod(**inputs)
2025-09-07T09:06:37.2789031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2789093Z     outputs = self.bert(
2025-09-07T09:06:37.2789339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2789411Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2789650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2789712Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2789930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2789998Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2790241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2790311Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2790551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2790611Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2790903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:06:37.2790997Z     self_outputs = self.self(
2025-09-07T09:06:37.2791235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2791298Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2791540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:06:37.2791660Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:06:37.2791663Z 
2025-09-07T09:06:37.2791760Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2791942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2792004Z     return mod(**inputs)
2025-09-07T09:06:37.2792280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2792345Z     outputs = self.bert(
2025-09-07T09:06:37.2792596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2792661Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2792940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2793003Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2793224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2793294Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2793538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:06:37.2793610Z     self_attention_outputs = self.attention(
2025-09-07T09:06:37.2793847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:06:37.2793913Z     return func(*args, **kwargs)
2025-09-07T09:06:37.2794150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:06:37.2794270Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:06:37.2794512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:06:37.2794584Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2794587Z 
2025-09-07T09:06:37.2794689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2794889Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2794948Z     return mod(**inputs)
2025-09-07T09:06:37.2795213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2795275Z     outputs = self.bert(
2025-09-07T09:06:37.2795526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2795590Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2795832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2795894Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2796107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2796181Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2796421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2796497Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2796798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2796903Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2797188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2797300Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2797546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:06:37.2797618Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2797622Z 
2025-09-07T09:06:37.2797720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2797911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2797965Z     return mod(**inputs)
2025-09-07T09:06:37.2798246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2798307Z     outputs = self.bert(
2025-09-07T09:06:37.2798553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2798616Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2798888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2798955Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2799166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2799245Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2799483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2799558Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2799818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2799883Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2800158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:06:37.2800266Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:06:37.2800508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:06:37.2800606Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:06:37.2800811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:06:37.2800878Z     return self.act(input)
2025-09-07T09:06:37.2800882Z 
2025-09-07T09:06:37.2800977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2801167Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2801221Z     return mod(**inputs)
2025-09-07T09:06:37.2801466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:06:37.2801525Z     outputs = self.bert(
2025-09-07T09:06:37.2801766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:06:37.2801832Z     encoder_outputs = self.encoder(
2025-09-07T09:06:37.2802068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:06:37.2802132Z     layer_outputs = layer_module(
2025-09-07T09:06:37.2802344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:06:37.2802446Z     return super().__call__(*args, **kwargs)
2025-09-07T09:06:37.2802723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:06:37.2802794Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:06:37.2803187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:06:37.2803251Z     return forward_fn(*input_tensors)
2025-09-07T09:06:37.2803544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:06:37.2803679Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:06:37.2803932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:06:37.2804010Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2804015Z 
2025-09-07T09:06:37.2804170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2804369Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2804426Z     return mod(**inputs)
2025-09-07T09:06:37.2804674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T09:06:37.2804813Z     prediction_scores = self.cls(sequence_output)
2025-09-07T09:06:37.2805061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T09:06:37.2805173Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T09:06:37.2805415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward
2025-09-07T09:06:37.2805492Z     hidden_states = self.transform(hidden_states)
2025-09-07T09:06:37.2805737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward
2025-09-07T09:06:37.2805811Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:06:37.2805814Z 
2025-09-07T09:06:37.2805909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:06:37.2806095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:06:37.2806155Z     return mod(**inputs)
2025-09-07T09:06:37.2806401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T09:06:37.2806481Z     prediction_scores = self.cls(sequence_output)
2025-09-07T09:06:37.2806723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T09:06:37.2806820Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T09:06:37.2807064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward
2025-09-07T09:06:37.2807147Z     hidden_states = self.decoder(hidden_states)
2025-09-07T09:06:37.2807150Z 
2025-09-07T09:06:44.6500829Z 
2025-09-07T09:06:44.7542001Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:06:44.8577182Z running benchmark:   3% 1/30 [00:00<00:03,  9.62it/s]
2025-09-07T09:06:44.9591551Z running benchmark:   7% 2/30 [00:00<00:02,  9.64it/s]
2025-09-07T09:06:45.0596912Z running benchmark:  10% 3/30 [00:00<00:02,  9.74it/s]
2025-09-07T09:06:45.2615495Z running benchmark:  13% 4/30 [00:00<00:02,  9.82it/s]
2025-09-07T09:06:45.3620721Z running benchmark:  20% 6/30 [00:00<00:02,  9.87it/s]
2025-09-07T09:06:45.5592098Z running benchmark:  23% 7/30 [00:00<00:02,  9.89it/s]
2025-09-07T09:06:45.6599851Z running benchmark:  30% 9/30 [00:00<00:02, 10.00it/s]
2025-09-07T09:06:45.7624109Z running benchmark:  33% 10/30 [00:01<00:02,  9.98it/s]
2025-09-07T09:06:45.8640471Z running benchmark:  37% 11/30 [00:01<00:01,  9.92it/s]
2025-09-07T09:06:45.9647631Z running benchmark:  40% 12/30 [00:01<00:01,  9.90it/s]
2025-09-07T09:06:46.1633387Z running benchmark:  43% 13/30 [00:01<00:01,  9.91it/s]
2025-09-07T09:06:46.3621346Z running benchmark:  50% 15/30 [00:01<00:01,  9.98it/s]
2025-09-07T09:06:46.4624427Z running benchmark:  57% 17/30 [00:01<00:01, 10.01it/s]
2025-09-07T09:06:46.5627552Z running benchmark:  60% 18/30 [00:01<00:01, 10.00it/s]
2025-09-07T09:06:46.6629503Z running benchmark:  63% 19/30 [00:01<00:01,  9.99it/s]
2025-09-07T09:06:46.7632739Z running benchmark:  67% 20/30 [00:02<00:01,  9.99it/s]
2025-09-07T09:06:46.8638607Z running benchmark:  70% 21/30 [00:02<00:00,  9.98it/s]
2025-09-07T09:06:46.9665722Z running benchmark:  73% 22/30 [00:02<00:00,  9.97it/s]
2025-09-07T09:06:47.0669593Z running benchmark:  77% 23/30 [00:02<00:00,  9.91it/s]
2025-09-07T09:06:47.1677593Z running benchmark:  80% 24/30 [00:02<00:00,  9.92it/s]
2025-09-07T09:06:47.3677137Z running benchmark:  83% 25/30 [00:02<00:00,  9.92it/s]
2025-09-07T09:06:47.4695176Z running benchmark:  90% 27/30 [00:02<00:00,  9.96it/s]
2025-09-07T09:06:47.5712177Z running benchmark:  93% 28/30 [00:02<00:00,  9.93it/s]
2025-09-07T09:06:47.6736442Z running benchmark:  97% 29/30 [00:02<00:00,  9.90it/s]
2025-09-07T09:06:47.6736827Z running benchmark: 100% 30/30 [00:03<00:00,  9.86it/s]
2025-09-07T09:06:47.6737148Z running benchmark: 100% 30/30 [00:03<00:00,  9.92it/s]
2025-09-07T09:06:47.7663153Z 1.180x
2025-09-07T09:06:47.7666640Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:06:50.1408952Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:06:50.1409834Z   import pynvml  # type: ignore[import]
2025-09-07T09:06:52.1670320Z 
2025-09-07T09:06:56.7573299Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:06:56.7573600Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:06:56.7893416Z cpu  eval  hf_Bert_large                      
2025-09-07T09:07:06.0473693Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0473972Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0474183Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0474419Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0474619Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0474835Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0475036Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0475234Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0475427Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0475621Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0475809Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0476000Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0476223Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0476426Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0476645Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0476828Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0477017Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0477208Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0477406Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0477594Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0477792Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0477978Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0478174Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0478361Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0478585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0478970Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0479590Z     return mod(**inputs)
2025-09-07T09:07:06.0480017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0480538Z     outputs = self.bert(
2025-09-07T09:07:06.0480918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0481312Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0481693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0482068Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0482426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0482799Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0483199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0483690Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0484092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0484473Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0484922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0485313Z     self_outputs = self.self(
2025-09-07T09:07:06.0485705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0486089Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0486465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0487002Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0487271Z 
2025-09-07T09:07:06.0487384Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0487750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0488076Z     return mod(**inputs)
2025-09-07T09:07:06.0488455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0488848Z     outputs = self.bert(
2025-09-07T09:07:06.0489200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0489574Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0489961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0490340Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0490695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0491071Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0491450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0491860Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0492266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0492655Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0493028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0493416Z     self_outputs = self.self(
2025-09-07T09:07:06.0493782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0494158Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0494614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0495048Z     self.key(current_states)
2025-09-07T09:07:06.0495169Z 
2025-09-07T09:07:06.0495274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0495637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0495971Z     return mod(**inputs)
2025-09-07T09:07:06.0496332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0496720Z     outputs = self.bert(
2025-09-07T09:07:06.0497070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0497533Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0497904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0498326Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0498681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0499042Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0499455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0499853Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0500224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0500590Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0500946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0501335Z     self_outputs = self.self(
2025-09-07T09:07:06.0501691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0502063Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0502422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0502798Z     self.value(current_states)
2025-09-07T09:07:06.0502910Z 
2025-09-07T09:07:06.0502992Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0503356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0503724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0504045Z     return mod(**inputs)
2025-09-07T09:07:06.0504411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0504799Z     outputs = self.bert(
2025-09-07T09:07:06.0505142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0505527Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0505902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0506279Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0506613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0506978Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0507351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0507740Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0508115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0508480Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0508918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0509354Z     self_outputs = self.self(
2025-09-07T09:07:06.0509721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0510099Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0510467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0510922Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0511117Z 
2025-09-07T09:07:06.0511218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0511578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0511901Z     return mod(**inputs)
2025-09-07T09:07:06.0512313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0512712Z     outputs = self.bert(
2025-09-07T09:07:06.0513083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0513462Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0513888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0514296Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0514643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0515012Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0515391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0515779Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0516164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0516537Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0516905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0517329Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0518227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0518616Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0518752Z 
2025-09-07T09:07:06.0518864Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0519217Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0519552Z     return mod(**inputs)
2025-09-07T09:07:06.0519912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0520298Z     outputs = self.bert(
2025-09-07T09:07:06.0520641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0521038Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0521424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0521809Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0522157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0522513Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0522898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0523291Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0523746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0524187Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0524603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0525085Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0525517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0525917Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0526059Z 
2025-09-07T09:07:06.0526160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0526523Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0526841Z     return mod(**inputs)
2025-09-07T09:07:06.0527241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0527627Z     outputs = self.bert(
2025-09-07T09:07:06.0527967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0528356Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0540392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0540855Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0541249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0541662Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0542096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0542543Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0542984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0543402Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0543839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0544318Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0544751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0545169Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0545543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0545892Z     return self.act(input)
2025-09-07T09:07:06.0546024Z 
2025-09-07T09:07:06.0546175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0546575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0546900Z     return mod(**inputs)
2025-09-07T09:07:06.0547258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0547634Z     outputs = self.bert(
2025-09-07T09:07:06.0547981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0548360Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0548725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0549106Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0549450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0549873Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0550293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0550676Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0551079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0551477Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0551887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0552359Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0552790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0553178Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0553322Z 
2025-09-07T09:07:06.0553463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0553824Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0554147Z     return mod(**inputs)
2025-09-07T09:07:06.0554509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0554920Z     outputs = self.bert(
2025-09-07T09:07:06.0555267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0555647Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0556013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0556385Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0556722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0557085Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0557456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0557839Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0558219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0558588Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0558950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0559319Z     self_outputs = self.self(
2025-09-07T09:07:06.0559674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0560038Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0560398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0560920Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0561184Z 
2025-09-07T09:07:06.0561286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0561649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0561964Z     return mod(**inputs)
2025-09-07T09:07:06.0562313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0562681Z     outputs = self.bert(
2025-09-07T09:07:06.0563019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0563396Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0563765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0564217Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0564550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0564919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0565294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0565681Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0566049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0566415Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0566774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0567143Z     self_outputs = self.self(
2025-09-07T09:07:06.0567530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0567896Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0568255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0568627Z     self.key(current_states)
2025-09-07T09:07:06.0568736Z 
2025-09-07T09:07:06.0568873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0569228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0569537Z     return mod(**inputs)
2025-09-07T09:07:06.0569886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0570257Z     outputs = self.bert(
2025-09-07T09:07:06.0570602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0570978Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0571354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0571731Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0572083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0572441Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0572815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0573199Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0573575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0573943Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0574305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0574674Z     self_outputs = self.self(
2025-09-07T09:07:06.0575025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0575390Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0575746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0576111Z     self.value(current_states)
2025-09-07T09:07:06.0576230Z 
2025-09-07T09:07:06.0576309Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0576542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0576899Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0577217Z     return mod(**inputs)
2025-09-07T09:07:06.0577629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0578133Z     outputs = self.bert(
2025-09-07T09:07:06.0578485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0578867Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0579235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0579611Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0579957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0580321Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0580697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0581079Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0582544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0582934Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0583295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0583664Z     self_outputs = self.self(
2025-09-07T09:07:06.0584069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0584442Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0584821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0585265Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0585449Z 
2025-09-07T09:07:06.0585552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0585912Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0586237Z     return mod(**inputs)
2025-09-07T09:07:06.0586599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0586978Z     outputs = self.bert(
2025-09-07T09:07:06.0587318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0587699Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0588072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0588452Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0588787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0589147Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0589528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0589912Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0590286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0590646Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0591006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0591434Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0591862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0592257Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0592392Z 
2025-09-07T09:07:06.0592499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0592916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0593285Z     return mod(**inputs)
2025-09-07T09:07:06.0593635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0594007Z     outputs = self.bert(
2025-09-07T09:07:06.0594346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0594723Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0595088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0595461Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0595799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0596144Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0596562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0596956Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0597364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0597793Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0598200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0598661Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0599087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0599480Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0599614Z 
2025-09-07T09:07:06.0599721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0600080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0600405Z     return mod(**inputs)
2025-09-07T09:07:06.0600761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0601135Z     outputs = self.bert(
2025-09-07T09:07:06.0601473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0601854Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0602220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0602588Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0602921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0603518Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0603897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0604295Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0604690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0605073Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0605472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0605918Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0606336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0606748Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0607215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0607600Z     return self.act(input)
2025-09-07T09:07:06.0607709Z 
2025-09-07T09:07:06.0607818Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0608169Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0608480Z     return mod(**inputs)
2025-09-07T09:07:06.0608827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0609193Z     outputs = self.bert(
2025-09-07T09:07:06.0609533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0609905Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0610266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0610687Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0611028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0611385Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0611755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0612179Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0612571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0612961Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0613375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0613845Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0614281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0614669Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0614803Z 
2025-09-07T09:07:06.0614901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0615250Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0615562Z     return mod(**inputs)
2025-09-07T09:07:06.0615907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0616275Z     outputs = self.bert(
2025-09-07T09:07:06.0616610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0616986Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0617354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0617782Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0618125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0618479Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0618858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0619243Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0619618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0619988Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0620345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0620711Z     self_outputs = self.self(
2025-09-07T09:07:06.0621111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0621522Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0621873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0622389Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0622652Z 
2025-09-07T09:07:06.0622753Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0623104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0623415Z     return mod(**inputs)
2025-09-07T09:07:06.0623783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0624176Z     outputs = self.bert(
2025-09-07T09:07:06.0624568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0624959Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0625323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0625692Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0626060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0626422Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0626799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0627175Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0627550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0627913Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0628258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0628624Z     self_outputs = self.self(
2025-09-07T09:07:06.0628974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0629328Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0629677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0630062Z     self.key(current_states)
2025-09-07T09:07:06.0630166Z 
2025-09-07T09:07:06.0630263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0630606Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0630944Z     return mod(**inputs)
2025-09-07T09:07:06.0631297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0631674Z     outputs = self.bert(
2025-09-07T09:07:06.0632015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0632411Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0632769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0633131Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0633477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0633842Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0634202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0634585Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0634996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0635405Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0635750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0636131Z     self_outputs = self.self(
2025-09-07T09:07:06.0636484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0636848Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0637191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0637551Z     self.value(current_states)
2025-09-07T09:07:06.0637667Z 
2025-09-07T09:07:06.0637750Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0637971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0638343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0638656Z     return mod(**inputs)
2025-09-07T09:07:06.0638996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0639365Z     outputs = self.bert(
2025-09-07T09:07:06.0639746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0640127Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0640482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0640868Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0641200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0641548Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0641916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0642309Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0642690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0643047Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0643413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0643786Z     self_outputs = self.self(
2025-09-07T09:07:06.0644127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0644497Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0644839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0645278Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0645459Z 
2025-09-07T09:07:06.0645556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0645892Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0646224Z     return mod(**inputs)
2025-09-07T09:07:06.0646582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0646945Z     outputs = self.bert(
2025-09-07T09:07:06.0647287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0647660Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0648032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0648397Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0648817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0649200Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0649575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0649949Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0650329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0650686Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0651057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0651496Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0651952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0652339Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0652475Z 
2025-09-07T09:07:06.0652574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0652917Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0653240Z     return mod(**inputs)
2025-09-07T09:07:06.0653632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0654012Z     outputs = self.bert(
2025-09-07T09:07:06.0654357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0654753Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0655127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0655496Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0655843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0656197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0656577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0656956Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0657341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0657812Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0658230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0658690Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0659104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0659485Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0659629Z 
2025-09-07T09:07:06.0659729Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0660080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0660397Z     return mod(**inputs)
2025-09-07T09:07:06.0660752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0661125Z     outputs = self.bert(
2025-09-07T09:07:06.0661471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0661851Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0662212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0662642Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0663015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0663366Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0663740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0664127Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0664511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0664900Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0665302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0665754Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0666207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0666624Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0666999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0667340Z     return self.act(input)
2025-09-07T09:07:06.0667460Z 
2025-09-07T09:07:06.0667597Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0667948Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0668262Z     return mod(**inputs)
2025-09-07T09:07:06.0668613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0668986Z     outputs = self.bert(
2025-09-07T09:07:06.0669330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0669707Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0670072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0670445Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0670779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0671136Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0671501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0671885Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0672278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0672668Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0673072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0673539Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0673976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0674384Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0674520Z 
2025-09-07T09:07:06.0674623Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0674966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0675275Z     return mod(**inputs)
2025-09-07T09:07:06.0675624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0676002Z     outputs = self.bert(
2025-09-07T09:07:06.0676345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0676785Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0677156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0677529Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0677873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0678225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0678598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0678987Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0679378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0679758Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0680163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0680552Z     self_outputs = self.self(
2025-09-07T09:07:06.0680908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0681274Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0681660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0682189Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0682458Z 
2025-09-07T09:07:06.0682558Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0682916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0683241Z     return mod(**inputs)
2025-09-07T09:07:06.0683600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0683977Z     outputs = self.bert(
2025-09-07T09:07:06.0684320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0684699Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0685070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0685438Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0685776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0686135Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0686511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0686895Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0687278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0687648Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0688004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0688373Z     self_outputs = self.self(
2025-09-07T09:07:06.0688715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0689078Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0689426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0689797Z     self.key(current_states)
2025-09-07T09:07:06.0689903Z 
2025-09-07T09:07:06.0690005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0690396Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0690749Z     return mod(**inputs)
2025-09-07T09:07:06.0691104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0691477Z     outputs = self.bert(
2025-09-07T09:07:06.0691817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0692200Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0692564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0692940Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0693276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0693629Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0694040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0694428Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0694796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0695162Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0695568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0695948Z     self_outputs = self.self(
2025-09-07T09:07:06.0696300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0696667Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0697015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0697457Z     self.value(current_states)
2025-09-07T09:07:06.0697577Z 
2025-09-07T09:07:06.0697654Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0697880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0698222Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0698537Z     return mod(**inputs)
2025-09-07T09:07:06.0698892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0699264Z     outputs = self.bert(
2025-09-07T09:07:06.0699604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0699980Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0700342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0700712Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0701046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0701405Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0701768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0702145Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0702515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0702883Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0703384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0703764Z     self_outputs = self.self(
2025-09-07T09:07:06.0704111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0704555Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0704973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0705404Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0705591Z 
2025-09-07T09:07:06.0705686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0706037Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0706357Z     return mod(**inputs)
2025-09-07T09:07:06.0706704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0707074Z     outputs = self.bert(
2025-09-07T09:07:06.0707420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0707801Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0708212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0708587Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0708924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0709326Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0709703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0710085Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0710449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0710816Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0711177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0711610Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0712030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0712424Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0712566Z 
2025-09-07T09:07:06.0712662Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0713013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0713329Z     return mod(**inputs)
2025-09-07T09:07:06.0713706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0714078Z     outputs = self.bert(
2025-09-07T09:07:06.0714441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0714822Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0715185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0715551Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0715886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0716239Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0716611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0716987Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0717387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0717783Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0718193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0718727Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0719140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0719521Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0719656Z 
2025-09-07T09:07:06.0719749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0720094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0720410Z     return mod(**inputs)
2025-09-07T09:07:06.0720748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0721120Z     outputs = self.bert(
2025-09-07T09:07:06.0721463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0721877Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0722241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0722612Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0722943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0723330Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0723705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0724087Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0724478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0724866Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0725270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0725714Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0726129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0726541Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0726917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0727260Z     return self.act(input)
2025-09-07T09:07:06.0727365Z 
2025-09-07T09:07:06.0727461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0727806Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0728120Z     return mod(**inputs)
2025-09-07T09:07:06.0728468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0728842Z     outputs = self.bert(
2025-09-07T09:07:06.0729180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0729563Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0729932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0730306Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0730633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0730983Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0731352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0731737Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0732180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0732601Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0733001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0733463Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0733894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0734278Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0734407Z 
2025-09-07T09:07:06.0734499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0734847Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0735156Z     return mod(**inputs)
2025-09-07T09:07:06.0735540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0735916Z     outputs = self.bert(
2025-09-07T09:07:06.0736263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0736636Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0737033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0737467Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0737796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0738149Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0738521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0738909Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0739283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0739648Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0740007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0740380Z     self_outputs = self.self(
2025-09-07T09:07:06.0740735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0741095Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0741446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0741965Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0742228Z 
2025-09-07T09:07:06.0742336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0742679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0742988Z     return mod(**inputs)
2025-09-07T09:07:06.0743341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0743710Z     outputs = self.bert(
2025-09-07T09:07:06.0744074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0744456Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0744813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0745188Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0745521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0745934Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0746380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0746766Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0747133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0747504Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0747860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0748224Z     self_outputs = self.self(
2025-09-07T09:07:06.0748576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0748937Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0749336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0749713Z     self.key(current_states)
2025-09-07T09:07:06.0749831Z 
2025-09-07T09:07:06.0749927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0750275Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0750587Z     return mod(**inputs)
2025-09-07T09:07:06.0750967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0751338Z     outputs = self.bert(
2025-09-07T09:07:06.0751684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0752065Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0752433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0752808Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0753135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0753487Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0753863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0754252Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0754616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0754986Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0755339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0755709Z     self_outputs = self.self(
2025-09-07T09:07:06.0756057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0756422Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0756774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0757151Z     self.value(current_states)
2025-09-07T09:07:06.0757262Z 
2025-09-07T09:07:06.0757345Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0757564Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0757906Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0758242Z     return mod(**inputs)
2025-09-07T09:07:06.0758591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0758962Z     outputs = self.bert(
2025-09-07T09:07:06.0759296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0759726Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0760143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0760521Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0760855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0761214Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0761586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0761973Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0762344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0762710Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0763103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0763489Z     self_outputs = self.self(
2025-09-07T09:07:06.0763845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0764209Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0764591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0765032Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0765225Z 
2025-09-07T09:07:06.0765324Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0765678Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0765993Z     return mod(**inputs)
2025-09-07T09:07:06.0766351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0766729Z     outputs = self.bert(
2025-09-07T09:07:06.0767078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0767463Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0767818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0768221Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0768559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0768909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0769276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0769657Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0770028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0770398Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0770759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0771188Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0771616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0772003Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0772143Z 
2025-09-07T09:07:06.0772251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0772613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0772928Z     return mod(**inputs)
2025-09-07T09:07:06.0773277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0773705Z     outputs = self.bert(
2025-09-07T09:07:06.0774089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0774464Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0774828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0775204Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0775544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0775899Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0776269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0776657Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0777094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0777579Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0777989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0778454Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0778907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0779303Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0779437Z 
2025-09-07T09:07:06.0779542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0779895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0780206Z     return mod(**inputs)
2025-09-07T09:07:06.0780558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0780936Z     outputs = self.bert(
2025-09-07T09:07:06.0781281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0781655Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0782021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0782394Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0782728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0783082Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0783452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0783836Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0784234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0784626Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0785022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0785476Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0785888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0786303Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0786677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0787006Z     return self.act(input)
2025-09-07T09:07:06.0787117Z 
2025-09-07T09:07:06.0787211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0787606Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0787957Z     return mod(**inputs)
2025-09-07T09:07:06.0788307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0788679Z     outputs = self.bert(
2025-09-07T09:07:06.0789030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0789411Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0789778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0790145Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0790483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0790836Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0791250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0791643Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0792032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0792420Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0792878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0793349Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0793785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0794163Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0794299Z 
2025-09-07T09:07:06.0794391Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0794746Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0795058Z     return mod(**inputs)
2025-09-07T09:07:06.0795400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0795777Z     outputs = self.bert(
2025-09-07T09:07:06.0796118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0796497Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0796854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0797230Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0797562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0797917Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0798285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0798678Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0799042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0799424Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0799800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0800173Z     self_outputs = self.self(
2025-09-07T09:07:06.0800532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0800897Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0801257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0801831Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0802157Z 
2025-09-07T09:07:06.0802261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0802604Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0802922Z     return mod(**inputs)
2025-09-07T09:07:06.0803408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0803794Z     outputs = self.bert(
2025-09-07T09:07:06.0804140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0804514Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0804896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0805345Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0805680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0806029Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0806396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0806829Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0807221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0807584Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0817462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0817907Z     self_outputs = self.self(
2025-09-07T09:07:06.0818320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0818713Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0819107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0819497Z     self.key(current_states)
2025-09-07T09:07:06.0819618Z 
2025-09-07T09:07:06.0819733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0820093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0820407Z     return mod(**inputs)
2025-09-07T09:07:06.0820767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0821141Z     outputs = self.bert(
2025-09-07T09:07:06.0821492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0821871Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0822247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0822636Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0822985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0823359Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0823764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0824159Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0824543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0824915Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0825279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0825772Z     self_outputs = self.self(
2025-09-07T09:07:06.0826185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0826555Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0826919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0827289Z     self.value(current_states)
2025-09-07T09:07:06.0827406Z 
2025-09-07T09:07:06.0827482Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0827715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0828072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0828381Z     return mod(**inputs)
2025-09-07T09:07:06.0828735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0829147Z     outputs = self.bert(
2025-09-07T09:07:06.0829490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0829861Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0830221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0830631Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0830968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0831322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0831695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0832067Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0832433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0832795Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0833148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0833516Z     self_outputs = self.self(
2025-09-07T09:07:06.0833868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0834224Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0834569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0834997Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0835176Z 
2025-09-07T09:07:06.0835274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0835621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0835932Z     return mod(**inputs)
2025-09-07T09:07:06.0836274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0836641Z     outputs = self.bert(
2025-09-07T09:07:06.0836972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0837353Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0837713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0838085Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0838419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0838772Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0839147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0839605Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0839972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0840330Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0840685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0841110Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0841527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0841909Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0842056Z 
2025-09-07T09:07:06.0842152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0842538Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0842867Z     return mod(**inputs)
2025-09-07T09:07:06.0843212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0843579Z     outputs = self.bert(
2025-09-07T09:07:06.0843951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0844329Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0844690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0845055Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0845382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0845732Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0846101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0846480Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0846870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0847264Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0847669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0848138Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0848561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0848952Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0849094Z 
2025-09-07T09:07:06.0849192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0849549Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0849872Z     return mod(**inputs)
2025-09-07T09:07:06.0850234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0850605Z     outputs = self.bert(
2025-09-07T09:07:06.0850951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0851327Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0851690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0852056Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0852391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0852750Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0853182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0853619Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0854021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0854417Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0854826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0855284Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0855710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0856128Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0856540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0856889Z     return self.act(input)
2025-09-07T09:07:06.0857002Z 
2025-09-07T09:07:06.0857108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0857499Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0857821Z     return mod(**inputs)
2025-09-07T09:07:06.0858212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0858599Z     outputs = self.bert(
2025-09-07T09:07:06.0858949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0859330Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0859700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0860087Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0860429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0860783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0861157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0861541Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0861935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0862331Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0862724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0863191Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0863632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0864017Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0864152Z 
2025-09-07T09:07:06.0864254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0864599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0864913Z     return mod(**inputs)
2025-09-07T09:07:06.0865261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0865638Z     outputs = self.bert(
2025-09-07T09:07:06.0865970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0866337Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0866700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0867132Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0867505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0867855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0868230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0868608Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0868976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0869341Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0869692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0870058Z     self_outputs = self.self(
2025-09-07T09:07:06.0870438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0870802Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0871150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0871664Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0871934Z 
2025-09-07T09:07:06.0872070Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0872424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0872484Z     return mod(**inputs)
2025-09-07T09:07:06.0872753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0872818Z     outputs = self.bert(
2025-09-07T09:07:06.0873063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0873132Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0873376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0873440Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0873659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0873728Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0873972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0874042Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0874278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0874344Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0874586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0874654Z     self_outputs = self.self(
2025-09-07T09:07:06.0874887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0874946Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0875190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0875253Z     self.key(current_states)
2025-09-07T09:07:06.0875256Z 
2025-09-07T09:07:06.0875350Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0875540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0875600Z     return mod(**inputs)
2025-09-07T09:07:06.0875846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0875940Z     outputs = self.bert(
2025-09-07T09:07:06.0876228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0876290Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0876534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0876597Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0876808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0876881Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0877121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0877193Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0877457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0877523Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0877768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0877830Z     self_outputs = self.self(
2025-09-07T09:07:06.0878099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0878159Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0878404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0878468Z     self.value(current_states)
2025-09-07T09:07:06.0878472Z 
2025-09-07T09:07:06.0878546Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0878643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0878831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0878895Z     return mod(**inputs)
2025-09-07T09:07:06.0879139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0879194Z     outputs = self.bert(
2025-09-07T09:07:06.0879446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0879509Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0879753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0879812Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0880024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0880093Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0880332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0880410Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0880643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0880705Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0880947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0881006Z     self_outputs = self.self(
2025-09-07T09:07:06.0881239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0881296Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0881539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0881661Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0881704Z 
2025-09-07T09:07:06.0881824Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0882011Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0882068Z     return mod(**inputs)
2025-09-07T09:07:06.0882329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0882389Z     outputs = self.bert(
2025-09-07T09:07:06.0882632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0882697Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0882941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0883012Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0883280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0883358Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0883600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0883671Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0883942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0884009Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0884253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0884373Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0884617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0884699Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0884704Z 
2025-09-07T09:07:06.0884794Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0884984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0885041Z     return mod(**inputs)
2025-09-07T09:07:06.0885289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0885349Z     outputs = self.bert(
2025-09-07T09:07:06.0885592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0885660Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0885903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0885967Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0886184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0886251Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0886497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0886571Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0886837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0886907Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0887187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0887303Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0887544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0887667Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0887700Z 
2025-09-07T09:07:06.0887789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0887978Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0888033Z     return mod(**inputs)
2025-09-07T09:07:06.0888279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0888340Z     outputs = self.bert(
2025-09-07T09:07:06.0888581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0888651Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0888897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0888958Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0889208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0889279Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0889526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0889600Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0889902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0889977Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0890251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0890362Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0890604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0890713Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0890921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0890984Z     return self.act(input)
2025-09-07T09:07:06.0890988Z 
2025-09-07T09:07:06.0891080Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0891267Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0891327Z     return mod(**inputs)
2025-09-07T09:07:06.0891572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0891631Z     outputs = self.bert(
2025-09-07T09:07:06.0891881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0891946Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0892189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0892250Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0892465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0892533Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0892774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0892847Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0893103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0893173Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0893446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0893636Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0893884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0893955Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0893958Z 
2025-09-07T09:07:06.0894055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0894238Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0894301Z     return mod(**inputs)
2025-09-07T09:07:06.0894544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0894601Z     outputs = self.bert(
2025-09-07T09:07:06.0894849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0894946Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0895196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0895258Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0895471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0895576Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0895821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0895899Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0896138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0896203Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0896450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0896517Z     self_outputs = self.self(
2025-09-07T09:07:06.0896755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0896815Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0897062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0897264Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0897268Z 
2025-09-07T09:07:06.0897361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0897624Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0897683Z     return mod(**inputs)
2025-09-07T09:07:06.0897938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0897996Z     outputs = self.bert(
2025-09-07T09:07:06.0898241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0898308Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0898548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0898618Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0898830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0898899Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0899141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0899210Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0899492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0899588Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0899832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0899894Z     self_outputs = self.self(
2025-09-07T09:07:06.0900131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0900196Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0900439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0900501Z     self.key(current_states)
2025-09-07T09:07:06.0900504Z 
2025-09-07T09:07:06.0900592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0900809Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0900873Z     return mod(**inputs)
2025-09-07T09:07:06.0901120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0901182Z     outputs = self.bert(
2025-09-07T09:07:06.0901459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0901526Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0901768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0901830Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0902047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0902114Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0902359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0902430Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0902662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0902724Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0902967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0903032Z     self_outputs = self.self(
2025-09-07T09:07:06.0903473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0903537Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0903792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0903854Z     self.value(current_states)
2025-09-07T09:07:06.0903861Z 
2025-09-07T09:07:06.0903940Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0904037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0904226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0904288Z     return mod(**inputs)
2025-09-07T09:07:06.0904535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0904595Z     outputs = self.bert(
2025-09-07T09:07:06.0904839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0904907Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0905151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0905212Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0905748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0905866Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0906111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0906182Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0906424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0906488Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0906728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0906793Z     self_outputs = self.self(
2025-09-07T09:07:06.0907031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0907098Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0907388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0907518Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0907521Z 
2025-09-07T09:07:06.0907621Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0907851Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0907912Z     return mod(**inputs)
2025-09-07T09:07:06.0908158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0908217Z     outputs = self.bert(
2025-09-07T09:07:06.0908464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0908529Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0908776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0908839Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0909053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0909124Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0909366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0909441Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0909676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0909733Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0909977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0910105Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0910349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0910424Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0910428Z 
2025-09-07T09:07:06.0910520Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0910705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0910761Z     return mod(**inputs)
2025-09-07T09:07:06.0911005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0911061Z     outputs = self.bert(
2025-09-07T09:07:06.0911308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0911369Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0911654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0911755Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0911968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0912043Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0912286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0912361Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0912633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0912708Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0913011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0913177Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0913430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0913502Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0913506Z 
2025-09-07T09:07:06.0913599Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0913825Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0913885Z     return mod(**inputs)
2025-09-07T09:07:06.0914137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0914192Z     outputs = self.bert(
2025-09-07T09:07:06.0914435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0914504Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0914745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0914813Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0915025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0915098Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0915339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0915408Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0915671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0915737Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0916016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0916123Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0916364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0916470Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0916681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0916748Z     return self.act(input)
2025-09-07T09:07:06.0916751Z 
2025-09-07T09:07:06.0916841Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0917029Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0917084Z     return mod(**inputs)
2025-09-07T09:07:06.0917327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0917460Z     outputs = self.bert(
2025-09-07T09:07:06.0917739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0917807Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0918047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0918110Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0918326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0918396Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0918639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0918711Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0918997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0919076Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0919348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0919475Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0919777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0919857Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0919860Z 
2025-09-07T09:07:06.0919953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0920144Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0920204Z     return mod(**inputs)
2025-09-07T09:07:06.0920454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0920516Z     outputs = self.bert(
2025-09-07T09:07:06.0920760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0920824Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0921075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0921139Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0921359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0921429Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0921670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0921750Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0921990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0922058Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0922301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0922367Z     self_outputs = self.self(
2025-09-07T09:07:06.0922605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0922664Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0922907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0923117Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0923120Z 
2025-09-07T09:07:06.0923216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0923443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0923535Z     return mod(**inputs)
2025-09-07T09:07:06.0923788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0923848Z     outputs = self.bert(
2025-09-07T09:07:06.0924099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0924161Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0924407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0924469Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0924684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0924755Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0925032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0925112Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0925355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0925417Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0925693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0925755Z     self_outputs = self.self(
2025-09-07T09:07:06.0925994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0926053Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0926294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0926365Z     self.key(current_states)
2025-09-07T09:07:06.0926368Z 
2025-09-07T09:07:06.0926462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0926651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0926706Z     return mod(**inputs)
2025-09-07T09:07:06.0926959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0927018Z     outputs = self.bert(
2025-09-07T09:07:06.0927261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0927327Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0927569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0927634Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0927848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0927914Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0928156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0928226Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0928464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0928521Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0928761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0928824Z     self_outputs = self.self(
2025-09-07T09:07:06.0929056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0929154Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0929397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0929501Z     self.value(current_states)
2025-09-07T09:07:06.0929504Z 
2025-09-07T09:07:06.0929579Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0929669Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0929858Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0929913Z     return mod(**inputs)
2025-09-07T09:07:06.0930162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0930217Z     outputs = self.bert(
2025-09-07T09:07:06.0930458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0930525Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0930804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0930869Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0931080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0931148Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0931422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0931494Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0931731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0931789Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0932031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0932095Z     self_outputs = self.self(
2025-09-07T09:07:06.0932327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0932390Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0932626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0932750Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0932753Z 
2025-09-07T09:07:06.0932848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0933030Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0933090Z     return mod(**inputs)
2025-09-07T09:07:06.0933335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0933396Z     outputs = self.bert(
2025-09-07T09:07:06.0933641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0933705Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0933946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0934006Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0934223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0934290Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0934532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0934600Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0934832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0934931Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0935222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0935343Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0935580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0935656Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0935659Z 
2025-09-07T09:07:06.0935757Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0935940Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0935998Z     return mod(**inputs)
2025-09-07T09:07:06.0936241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0936301Z     outputs = self.bert(
2025-09-07T09:07:06.0936577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0936644Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0936888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0936948Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0937193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0937260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0937580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0937656Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0937915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0937989Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0938267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0938379Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0938623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0938694Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0938697Z 
2025-09-07T09:07:06.0938791Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0938975Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0939034Z     return mod(**inputs)
2025-09-07T09:07:06.0939278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0939340Z     outputs = self.bert(
2025-09-07T09:07:06.0939583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0939646Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0939888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0939949Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0940160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0940228Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0940470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0940543Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0940797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0940937Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0941212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0941315Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0941566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0941671Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0941881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0941946Z     return self.act(input)
2025-09-07T09:07:06.0941950Z 
2025-09-07T09:07:06.0942045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0942233Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0942328Z     return mod(**inputs)
2025-09-07T09:07:06.0942585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0942646Z     outputs = self.bert(
2025-09-07T09:07:06.0942888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0942995Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0943243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0943312Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0943530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0943603Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0943849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0943927Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0944186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0944252Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0944535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0944659Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0944898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0944970Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0944974Z 
2025-09-07T09:07:06.0945065Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0945254Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0945314Z     return mod(**inputs)
2025-09-07T09:07:06.0945560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0945623Z     outputs = self.bert(
2025-09-07T09:07:06.0945869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0945934Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0946173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0946236Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0946447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0946514Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0946797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0946902Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0947144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0947205Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0947447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0947513Z     self_outputs = self.self(
2025-09-07T09:07:06.0947748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0947810Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0948049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0948290Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0948295Z 
2025-09-07T09:07:06.0948386Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0948569Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0948628Z     return mod(**inputs)
2025-09-07T09:07:06.0948904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0948964Z     outputs = self.bert(
2025-09-07T09:07:06.0949207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0949269Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0949510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0949575Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0949789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0949857Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0950095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0950168Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0950403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0950464Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0950702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0950767Z     self_outputs = self.self(
2025-09-07T09:07:06.0951002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0951064Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0951309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0951373Z     self.key(current_states)
2025-09-07T09:07:06.0951377Z 
2025-09-07T09:07:06.0951474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0951660Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0951721Z     return mod(**inputs)
2025-09-07T09:07:06.0951968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0952025Z     outputs = self.bert(
2025-09-07T09:07:06.0952273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0952336Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0952614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0952715Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0952928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0953000Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0953244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0953321Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0953557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0953618Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0953862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0953961Z     self_outputs = self.self(
2025-09-07T09:07:06.0954201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0954261Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0954504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0954609Z     self.value(current_states)
2025-09-07T09:07:06.0954613Z 
2025-09-07T09:07:06.0954687Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0954783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0954966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0955023Z     return mod(**inputs)
2025-09-07T09:07:06.0955281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0955340Z     outputs = self.bert(
2025-09-07T09:07:06.0955587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0955652Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0955894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0955960Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0956174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0956245Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0956485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0956560Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0956795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0956857Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0957104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0957163Z     self_outputs = self.self(
2025-09-07T09:07:06.0957401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0957462Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0957703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0957828Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0957832Z 
2025-09-07T09:07:06.0957934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0958120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0958230Z     return mod(**inputs)
2025-09-07T09:07:06.0958484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0958577Z     outputs = self.bert(
2025-09-07T09:07:06.0958821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0958889Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0959133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0959196Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0959405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0959473Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0959715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0959821Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0960066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0960125Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0960366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0960523Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0960771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0960847Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0960851Z 
2025-09-07T09:07:06.0960942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0961129Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0961195Z     return mod(**inputs)
2025-09-07T09:07:06.0961458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0961523Z     outputs = self.bert(
2025-09-07T09:07:06.0961764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0961832Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0962071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0962136Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0962357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0962424Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0962667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0962742Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0963001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0963073Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0963351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0963466Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0963704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0963778Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0963782Z 
2025-09-07T09:07:06.0963872Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0964057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0964159Z     return mod(**inputs)
2025-09-07T09:07:06.0964438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0964503Z     outputs = self.bert(
2025-09-07T09:07:06.0964746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0964813Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0965058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0965121Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0965338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0965403Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0965678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0965754Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0966014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0966084Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0966387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0966498Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0966740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0966844Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0967057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0967125Z     return self.act(input)
2025-09-07T09:07:06.0967129Z 
2025-09-07T09:07:06.0967226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0967410Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0967471Z     return mod(**inputs)
2025-09-07T09:07:06.0967725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0967783Z     outputs = self.bert(
2025-09-07T09:07:06.0968032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0968094Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0968342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0968405Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0968620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0968696Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0968939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0969018Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0969274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0969340Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0969619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0969742Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0969989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0970099Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0970131Z 
2025-09-07T09:07:06.0970231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0970415Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0970473Z     return mod(**inputs)
2025-09-07T09:07:06.0970728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0970787Z     outputs = self.bert(
2025-09-07T09:07:06.0971039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0971102Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0971346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0971417Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0971664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0971741Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0971984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0972056Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0972336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0972400Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0972647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0972708Z     self_outputs = self.self(
2025-09-07T09:07:06.0972952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0973016Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0973259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0973464Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0973467Z 
2025-09-07T09:07:06.0973562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0973754Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0973813Z     return mod(**inputs)
2025-09-07T09:07:06.0974058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0974121Z     outputs = self.bert(
2025-09-07T09:07:06.0974367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0974439Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0974681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0974752Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0974963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0975028Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0975269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0975338Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0975574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0975632Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0975872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0975969Z     self_outputs = self.self(
2025-09-07T09:07:06.0976234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0976295Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0976535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.0976599Z     self.key(current_states)
2025-09-07T09:07:06.0976603Z 
2025-09-07T09:07:06.0976694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0976878Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0976937Z     return mod(**inputs)
2025-09-07T09:07:06.0977179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0977241Z     outputs = self.bert(
2025-09-07T09:07:06.0977551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0977617Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0977859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0977919Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0978330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0978398Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0978639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0978712Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0978945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0979009Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0979248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0979310Z     self_outputs = self.self(
2025-09-07T09:07:06.0979546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0979607Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0979850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.0979911Z     self.value(current_states)
2025-09-07T09:07:06.0979915Z 
2025-09-07T09:07:06.0979990Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.0980079Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0980261Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0980324Z     return mod(**inputs)
2025-09-07T09:07:06.0980567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0980627Z     outputs = self.bert(
2025-09-07T09:07:06.0980880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0980948Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0981194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0981257Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0981478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0981545Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0981785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0981910Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0982190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0982258Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0982502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0982569Z     self_outputs = self.self(
2025-09-07T09:07:06.0982813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0982876Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0983123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.0983243Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.0983250Z 
2025-09-07T09:07:06.0983382Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0983569Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0983629Z     return mod(**inputs)
2025-09-07T09:07:06.0983881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0983973Z     outputs = self.bert(
2025-09-07T09:07:06.0984226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0984291Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0984535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0984604Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0984818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0984894Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0985139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0985212Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0985455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0985515Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0985760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.0985878Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.0986125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.0986197Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0986202Z 
2025-09-07T09:07:06.0986296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0986488Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0986547Z     return mod(**inputs)
2025-09-07T09:07:06.0986799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0986861Z     outputs = self.bert(
2025-09-07T09:07:06.0987111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0987185Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0987423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0987488Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0987702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0987814Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0988090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0988163Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0988433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0988500Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0988784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0988893Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0989138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.0989214Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0989251Z 
2025-09-07T09:07:06.0989344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0989533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0989590Z     return mod(**inputs)
2025-09-07T09:07:06.0989870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0989934Z     outputs = self.bert(
2025-09-07T09:07:06.0990178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0990249Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0990488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0990556Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0990770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0990842Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0991084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0991153Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0991419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0991484Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0991758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.0991876Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.0992117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.0992224Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.0992430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.0992495Z     return self.act(input)
2025-09-07T09:07:06.0992499Z 
2025-09-07T09:07:06.0992588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0992774Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0992836Z     return mod(**inputs)
2025-09-07T09:07:06.0993079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0993139Z     outputs = self.bert(
2025-09-07T09:07:06.0993382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0993446Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0993737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0993836Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0994053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0994121Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0994367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.0994443Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.0994697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.0994766Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.0995034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.0995200Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.0995446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.0995520Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.0995523Z 
2025-09-07T09:07:06.0995619Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0995836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0995899Z     return mod(**inputs)
2025-09-07T09:07:06.0996145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0996205Z     outputs = self.bert(
2025-09-07T09:07:06.0996453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0996520Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.0996765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.0996828Z     layer_outputs = layer_module(
2025-09-07T09:07:06.0997043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.0997112Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.0997354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.0997435Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.0997674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0997740Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0997978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.0998043Z     self_outputs = self.self(
2025-09-07T09:07:06.0998284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.0998344Z     return func(*args, **kwargs)
2025-09-07T09:07:06.0998588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.0998798Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.0998802Z 
2025-09-07T09:07:06.0998901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.0999087Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.0999149Z     return mod(**inputs)
2025-09-07T09:07:06.0999404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.0999513Z     outputs = self.bert(
2025-09-07T09:07:06.0999796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.0999864Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1000104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1000171Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1000383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1000453Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1000693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1000765Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1001054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1001121Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1001387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1001449Z     self_outputs = self.self(
2025-09-07T09:07:06.1001719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1001789Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1002034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1002104Z     self.key(current_states)
2025-09-07T09:07:06.1002107Z 
2025-09-07T09:07:06.1002200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1002395Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1002455Z     return mod(**inputs)
2025-09-07T09:07:06.1002702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1002768Z     outputs = self.bert(
2025-09-07T09:07:06.1003014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1003231Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1003473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1003533Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1003747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1003812Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1004054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1004127Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1004363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1004422Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1004663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1004727Z     self_outputs = self.self(
2025-09-07T09:07:06.1004958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1005018Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1005255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1005317Z     self.value(current_states)
2025-09-07T09:07:06.1005320Z 
2025-09-07T09:07:06.1005395Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1005546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1005802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1005862Z     return mod(**inputs)
2025-09-07T09:07:06.1006115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1006180Z     outputs = self.bert(
2025-09-07T09:07:06.1006426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1006497Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1006741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1006804Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1007070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1007145Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1007390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1007460Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1007793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1007858Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1008103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1008172Z     self_outputs = self.self(
2025-09-07T09:07:06.1008410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1008477Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1008723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1008849Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1008854Z 
2025-09-07T09:07:06.1008956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1009144Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1009206Z     return mod(**inputs)
2025-09-07T09:07:06.1009450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1009509Z     outputs = self.bert(
2025-09-07T09:07:06.1009759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1009822Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1010065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1010128Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1010344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1010410Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1010651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1010725Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1010959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1011027Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1011270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1011386Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1011676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1011782Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1011785Z 
2025-09-07T09:07:06.1011880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1012070Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1012131Z     return mod(**inputs)
2025-09-07T09:07:06.1012378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1012438Z     outputs = self.bert(
2025-09-07T09:07:06.1012687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1012751Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1013028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1013095Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1013311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1013384Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1013660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1013739Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1013998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1014067Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1014347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1014457Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1014703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1014778Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1014782Z 
2025-09-07T09:07:06.1014877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1015065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1015122Z     return mod(**inputs)
2025-09-07T09:07:06.1015373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1015433Z     outputs = self.bert(
2025-09-07T09:07:06.1015678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1015740Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1015981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1016047Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1016264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1016334Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1016576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1016648Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1016909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1016975Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1017255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1017438Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1017720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1017820Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1018030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1018100Z     return self.act(input)
2025-09-07T09:07:06.1018104Z 
2025-09-07T09:07:06.1018196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1018387Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1018445Z     return mod(**inputs)
2025-09-07T09:07:06.1018692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1018754Z     outputs = self.bert(
2025-09-07T09:07:06.1019037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1019113Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1019356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1019426Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1019676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1019749Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1020001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1020073Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1020336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1020404Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1020678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1020807Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1021049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1021129Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1021133Z 
2025-09-07T09:07:06.1021221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1021414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1021472Z     return mod(**inputs)
2025-09-07T09:07:06.1021717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1021788Z     outputs = self.bert(
2025-09-07T09:07:06.1022027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1022096Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1022337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1022399Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1022617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1022685Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1022926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1022995Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1023234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1023340Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1023630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1023699Z     self_outputs = self.self(
2025-09-07T09:07:06.1023939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1024006Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1024249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1024450Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1024453Z 
2025-09-07T09:07:06.1024552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1024771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1024840Z     return mod(**inputs)
2025-09-07T09:07:06.1025085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1025145Z     outputs = self.bert(
2025-09-07T09:07:06.1025394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1025493Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1025743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1025806Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1026018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1026091Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1026333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1026410Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1026647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1026708Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1026948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1027011Z     self_outputs = self.self(
2025-09-07T09:07:06.1027249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1027309Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1027553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1027615Z     self.key(current_states)
2025-09-07T09:07:06.1027621Z 
2025-09-07T09:07:06.1027711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1027903Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1027963Z     return mod(**inputs)
2025-09-07T09:07:06.1028224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1028287Z     outputs = self.bert(
2025-09-07T09:07:06.1028535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1028609Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1028854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1028924Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1029144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1029264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1029542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1029619Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1029873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1029936Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1030188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1030255Z     self_outputs = self.self(
2025-09-07T09:07:06.1030493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1030561Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1030835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1030910Z     self.value(current_states)
2025-09-07T09:07:06.1030913Z 
2025-09-07T09:07:06.1030988Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1031089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1031324Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1031385Z     return mod(**inputs)
2025-09-07T09:07:06.1031636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1031695Z     outputs = self.bert(
2025-09-07T09:07:06.1031948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1032013Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1032254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1032324Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1032539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1032611Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1032853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1032923Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1033162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1033223Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1033466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1033525Z     self_outputs = self.self(
2025-09-07T09:07:06.1033761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1033829Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1034069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1034200Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1034205Z 
2025-09-07T09:07:06.1034299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1034495Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1034552Z     return mod(**inputs)
2025-09-07T09:07:06.1034798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1034866Z     outputs = self.bert(
2025-09-07T09:07:06.1035112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1035263Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1035506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1035572Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1035793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1035861Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1036104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1036173Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1036407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1036474Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1036749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1036878Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1037119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1037239Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1037242Z 
2025-09-07T09:07:06.1037344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1037534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1037598Z     return mod(**inputs)
2025-09-07T09:07:06.1037845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1037906Z     outputs = self.bert(
2025-09-07T09:07:06.1038153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1038222Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1038466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1038529Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1038744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1038809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1039053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1039128Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1039390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1039470Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1039752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1039870Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1040109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1040184Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1040188Z 
2025-09-07T09:07:06.1040284Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1040470Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1040531Z     return mod(**inputs)
2025-09-07T09:07:06.1040777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1040835Z     outputs = self.bert(
2025-09-07T09:07:06.1041131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1041229Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1041476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1041539Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1041758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1041826Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1042068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1042141Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1042398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1042500Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1042779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1042886Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1043158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1043263Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1043473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1043534Z     return self.act(input)
2025-09-07T09:07:06.1043538Z 
2025-09-07T09:07:06.1043630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1043815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1043872Z     return mod(**inputs)
2025-09-07T09:07:06.1044117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1044174Z     outputs = self.bert(
2025-09-07T09:07:06.1044417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1044481Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1044724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1044787Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1044998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1045066Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1045306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1045379Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1045633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1045699Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1045974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1046100Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1046340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1046408Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1046411Z 
2025-09-07T09:07:06.1046498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1046686Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1046797Z     return mod(**inputs)
2025-09-07T09:07:06.1047079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1047138Z     outputs = self.bert(
2025-09-07T09:07:06.1047382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1047450Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1047689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1047754Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1047965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1048034Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1048315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1048391Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1048656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1048717Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1049000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1049066Z     self_outputs = self.self(
2025-09-07T09:07:06.1049306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1049372Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1049616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1049829Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1049834Z 
2025-09-07T09:07:06.1049925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1050114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1050180Z     return mod(**inputs)
2025-09-07T09:07:06.1050432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1050502Z     outputs = self.bert(
2025-09-07T09:07:06.1050749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1050820Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1051061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1051126Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1058683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1058800Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1059100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1059189Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1059465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1059532Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1059804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1059873Z     self_outputs = self.self(
2025-09-07T09:07:06.1060132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1060196Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1060530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1060636Z     self.key(current_states)
2025-09-07T09:07:06.1060641Z 
2025-09-07T09:07:06.1060744Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1060949Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1061015Z     return mod(**inputs)
2025-09-07T09:07:06.1061274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1061336Z     outputs = self.bert(
2025-09-07T09:07:06.1061585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1061665Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1062078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1062150Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1062375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1062454Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1062739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1062815Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1063057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1063124Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1063366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1063433Z     self_outputs = self.self(
2025-09-07T09:07:06.1063674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1063739Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1063980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1064044Z     self.value(current_states)
2025-09-07T09:07:06.1064048Z 
2025-09-07T09:07:06.1064130Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1064230Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1064424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1064486Z     return mod(**inputs)
2025-09-07T09:07:06.1064736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1064796Z     outputs = self.bert(
2025-09-07T09:07:06.1065041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1065112Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1065353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1065414Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1065637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1065706Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1065952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1066025Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1066276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1066342Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1066624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1066722Z     self_outputs = self.self(
2025-09-07T09:07:06.1066956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1067018Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1067265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1067392Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1067396Z 
2025-09-07T09:07:06.1067492Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1067683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1067743Z     return mod(**inputs)
2025-09-07T09:07:06.1068026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1068092Z     outputs = self.bert(
2025-09-07T09:07:06.1068337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1068403Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1068684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1068750Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1068968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1069047Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1069289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1069369Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1069610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1069677Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1069921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1070048Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1070295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1070376Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1070380Z 
2025-09-07T09:07:06.1070495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1070698Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1070758Z     return mod(**inputs)
2025-09-07T09:07:06.1071016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1071078Z     outputs = self.bert(
2025-09-07T09:07:06.1071327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1071398Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1071642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1071708Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1071923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1071996Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1072237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1072354Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1072622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1072749Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1073035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1073154Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1073402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1073475Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1073478Z 
2025-09-07T09:07:06.1073574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1073770Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1073830Z     return mod(**inputs)
2025-09-07T09:07:06.1074117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1074179Z     outputs = self.bert(
2025-09-07T09:07:06.1074431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1074495Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1074768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1074842Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1075060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1075134Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1075376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1075454Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1075717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1075786Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1076066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1076173Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1076413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1076524Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1076735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1076801Z     return self.act(input)
2025-09-07T09:07:06.1076806Z 
2025-09-07T09:07:06.1076898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1077090Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1077149Z     return mod(**inputs)
2025-09-07T09:07:06.1077392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1077453Z     outputs = self.bert(
2025-09-07T09:07:06.1077696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1077761Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1077998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1078058Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1078272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1078381Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1078656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1078730Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1078989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1079063Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1079333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1079467Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1079711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1079788Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1079832Z 
2025-09-07T09:07:06.1079922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1080112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1080179Z     return mod(**inputs)
2025-09-07T09:07:06.1080455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1080522Z     outputs = self.bert(
2025-09-07T09:07:06.1080766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1080831Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1081079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1081142Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1081360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1081431Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1081678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1081753Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1081992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1082061Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1082301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1082366Z     self_outputs = self.self(
2025-09-07T09:07:06.1082601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1082659Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1082904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1083112Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1083116Z 
2025-09-07T09:07:06.1083209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1083397Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1083457Z     return mod(**inputs)
2025-09-07T09:07:06.1083701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1083759Z     outputs = self.bert(
2025-09-07T09:07:06.1084002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1084065Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1084348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1084441Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1084654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1084723Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1084963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1085038Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1085271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1085329Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1085568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1085632Z     self_outputs = self.self(
2025-09-07T09:07:06.1085898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1085960Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1086204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1086267Z     self.key(current_states)
2025-09-07T09:07:06.1086303Z 
2025-09-07T09:07:06.1086395Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1086581Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1086640Z     return mod(**inputs)
2025-09-07T09:07:06.1086890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1086948Z     outputs = self.bert(
2025-09-07T09:07:06.1087193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1087266Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1087507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1087572Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1087786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1087855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1088100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1088172Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1088412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1088472Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1088717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1088782Z     self_outputs = self.self(
2025-09-07T09:07:06.1089015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1089075Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1089315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1089381Z     self.value(current_states)
2025-09-07T09:07:06.1089384Z 
2025-09-07T09:07:06.1089456Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1089543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1089729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1089783Z     return mod(**inputs)
2025-09-07T09:07:06.1090071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1090161Z     outputs = self.bert(
2025-09-07T09:07:06.1090404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1090473Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1090713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1090775Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1090987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1091053Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1091300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1091371Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1091640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1091702Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1091946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1092043Z     self_outputs = self.self(
2025-09-07T09:07:06.1092283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1092348Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1092586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1092712Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1092715Z 
2025-09-07T09:07:06.1092953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1093145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1093203Z     return mod(**inputs)
2025-09-07T09:07:06.1093448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1093508Z     outputs = self.bert(
2025-09-07T09:07:06.1093753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1093819Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1094060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1094120Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1094335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1094402Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1094645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1094715Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1094946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1095009Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1095248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1095372Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1095610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1095685Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1095689Z 
2025-09-07T09:07:06.1095776Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1096013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1096108Z     return mod(**inputs)
2025-09-07T09:07:06.1096354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1096413Z     outputs = self.bert(
2025-09-07T09:07:06.1096659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1096721Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1096967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1097032Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1097248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1097347Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1097624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1097706Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1097967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1098073Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1098352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1098466Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1098708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1098777Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1098782Z 
2025-09-07T09:07:06.1098877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1099062Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1099120Z     return mod(**inputs)
2025-09-07T09:07:06.1099362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1099420Z     outputs = self.bert(
2025-09-07T09:07:06.1099671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1099733Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1099977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1100036Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1100247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1100321Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1100561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1100636Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1100900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1100967Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1101242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1101345Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1101585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1101687Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1101938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1102039Z     return self.act(input)
2025-09-07T09:07:06.1102043Z 
2025-09-07T09:07:06.1102135Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1102321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1102378Z     return mod(**inputs)
2025-09-07T09:07:06.1102622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1102684Z     outputs = self.bert(
2025-09-07T09:07:06.1102938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1103002Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1103520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1103593Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1103828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1103903Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1104212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1104293Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1104559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1104628Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1104913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1105045Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1105298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1105375Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1105379Z 
2025-09-07T09:07:06.1105478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1105680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1105745Z     return mod(**inputs)
2025-09-07T09:07:06.1106001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1106063Z     outputs = self.bert(
2025-09-07T09:07:06.1106306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1106372Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1106622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1106688Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1106913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1106984Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1107233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1107306Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1107550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1107617Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1107857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1108017Z     self_outputs = self.self(
2025-09-07T09:07:06.1108257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1108366Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1108614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1108822Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1108826Z 
2025-09-07T09:07:06.1108922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1109114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1109174Z     return mod(**inputs)
2025-09-07T09:07:06.1109421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1109477Z     outputs = self.bert(
2025-09-07T09:07:06.1109761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1109829Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1110075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1110138Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1110384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1110456Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1110696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1110770Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1111004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1111068Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1111313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1111375Z     self_outputs = self.self(
2025-09-07T09:07:06.1111616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1111678Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1111918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1111988Z     self.key(current_states)
2025-09-07T09:07:06.1111991Z 
2025-09-07T09:07:06.1112085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1112277Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1112334Z     return mod(**inputs)
2025-09-07T09:07:06.1112586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1112645Z     outputs = self.bert(
2025-09-07T09:07:06.1112887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1112953Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1113193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1113259Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1113472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1113542Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1113785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1113901Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1114140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1114232Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1114474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1114537Z     self_outputs = self.self(
2025-09-07T09:07:06.1114771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1114833Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1115071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1115136Z     self.value(current_states)
2025-09-07T09:07:06.1115139Z 
2025-09-07T09:07:06.1115212Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1115301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1115520Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1115579Z     return mod(**inputs)
2025-09-07T09:07:06.1115828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1115883Z     outputs = self.bert(
2025-09-07T09:07:06.1116175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1116244Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1116484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1116548Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1116759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1116828Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1117067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1117139Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1117372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1117431Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1117671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1117730Z     self_outputs = self.self(
2025-09-07T09:07:06.1117962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1118022Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1118263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1118390Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1118395Z 
2025-09-07T09:07:06.1118483Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1118666Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1118724Z     return mod(**inputs)
2025-09-07T09:07:06.1118970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1119032Z     outputs = self.bert(
2025-09-07T09:07:06.1119276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1119337Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1119578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1119685Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1119899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1120001Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1120243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1120313Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1120547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1120607Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1120846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1120966Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1121238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1121318Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1121321Z 
2025-09-07T09:07:06.1121412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1121592Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1121651Z     return mod(**inputs)
2025-09-07T09:07:06.1121929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1121990Z     outputs = self.bert(
2025-09-07T09:07:06.1122234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1122296Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1122537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1122599Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1122814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1122879Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1123116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1123196Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1123458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1123529Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1123805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1123918Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1124163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1124235Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1124238Z 
2025-09-07T09:07:06.1124330Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1124512Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1124572Z     return mod(**inputs)
2025-09-07T09:07:06.1124814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1124872Z     outputs = self.bert(
2025-09-07T09:07:06.1125118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1125182Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1125423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1125525Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1125769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1125840Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1126081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1126155Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1126410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1126481Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1126751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1126854Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1127131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1127238Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1127448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1127510Z     return self.act(input)
2025-09-07T09:07:06.1127544Z 
2025-09-07T09:07:06.1127635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1127820Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1127878Z     return mod(**inputs)
2025-09-07T09:07:06.1128129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1128185Z     outputs = self.bert(
2025-09-07T09:07:06.1128429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1128498Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1128739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1128801Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1129014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1129082Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1129319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1129389Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1129645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1129709Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1129988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1130114Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1130356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1130435Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1130439Z 
2025-09-07T09:07:06.1130531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1130724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1130779Z     return mod(**inputs)
2025-09-07T09:07:06.1131030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1131087Z     outputs = self.bert(
2025-09-07T09:07:06.1131368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1131472Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1131715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1131783Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1131998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1132068Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1132314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1132384Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1132625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1132721Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1132968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1133035Z     self_outputs = self.self(
2025-09-07T09:07:06.1133269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1133363Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1133605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1133810Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1133814Z 
2025-09-07T09:07:06.1133905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1134090Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1134154Z     return mod(**inputs)
2025-09-07T09:07:06.1134404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1134468Z     outputs = self.bert(
2025-09-07T09:07:06.1134710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1134772Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1135020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1135081Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1135301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1135368Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1135609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1135685Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1135919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1135980Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1136217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1136280Z     self_outputs = self.self(
2025-09-07T09:07:06.1136512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1136570Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1136810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1136871Z     self.key(current_states)
2025-09-07T09:07:06.1136874Z 
2025-09-07T09:07:06.1136964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1137211Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1137546Z     return mod(**inputs)
2025-09-07T09:07:06.1137792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1137850Z     outputs = self.bert(
2025-09-07T09:07:06.1138098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1138161Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1138401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1138464Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1138675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1138802Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1139047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1139122Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1139359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1139455Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1139701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1139767Z     self_outputs = self.self(
2025-09-07T09:07:06.1140002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1140059Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1140299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1140370Z     self.value(current_states)
2025-09-07T09:07:06.1140375Z 
2025-09-07T09:07:06.1140449Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1140544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1140727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1140782Z     return mod(**inputs)
2025-09-07T09:07:06.1141027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1141082Z     outputs = self.bert(
2025-09-07T09:07:06.1141324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1141385Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1141626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1141690Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1141903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1141972Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1142213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1142286Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1142519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1142577Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1142819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1142878Z     self_outputs = self.self(
2025-09-07T09:07:06.1143118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1143242Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1143480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1143606Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1143610Z 
2025-09-07T09:07:06.1143704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1143892Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1143947Z     return mod(**inputs)
2025-09-07T09:07:06.1144196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1144254Z     outputs = self.bert(
2025-09-07T09:07:06.1144496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1144602Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1144849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1144917Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1145128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1145226Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1145475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1145545Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1145782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1145843Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1146085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1146211Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1146456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1146536Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1146539Z 
2025-09-07T09:07:06.1146628Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1146813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1146868Z     return mod(**inputs)
2025-09-07T09:07:06.1147111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1147168Z     outputs = self.bert(
2025-09-07T09:07:06.1147411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1147478Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1147719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1147780Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1147991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1148060Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1148305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1148378Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1148644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1148713Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1149037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1149185Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1149426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1149502Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1149507Z 
2025-09-07T09:07:06.1149598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1149786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1149847Z     return mod(**inputs)
2025-09-07T09:07:06.1150104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1150163Z     outputs = self.bert(
2025-09-07T09:07:06.1150443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1150513Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1150758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1150819Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1151071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1151142Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1151386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1151457Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1151715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1151783Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1152063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1152175Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1152415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1152516Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1152730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1152791Z     return self.act(input)
2025-09-07T09:07:06.1152794Z 
2025-09-07T09:07:06.1152888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1153074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1153135Z     return mod(**inputs)
2025-09-07T09:07:06.1153384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1153444Z     outputs = self.bert(
2025-09-07T09:07:06.1153689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1153751Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1153999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1154060Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1154273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1154343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1154582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1154657Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1154953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1155052Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1155334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1155460Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1155708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1155778Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1155781Z 
2025-09-07T09:07:06.1155872Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1156056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1156111Z     return mod(**inputs)
2025-09-07T09:07:06.1156393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1156454Z     outputs = self.bert(
2025-09-07T09:07:06.1156701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1156763Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1157037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1157113Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1157327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1157395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1157633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1157713Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1157956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1158016Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1158259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1158321Z     self_outputs = self.self(
2025-09-07T09:07:06.1158562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1158620Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1158861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1159066Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1159071Z 
2025-09-07T09:07:06.1159162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1159353Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1159407Z     return mod(**inputs)
2025-09-07T09:07:06.1159650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1159715Z     outputs = self.bert(
2025-09-07T09:07:06.1159959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1160025Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1160263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1160327Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1160539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1160645Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1160945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1161021Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1161268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1161332Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1161572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1161637Z     self_outputs = self.self(
2025-09-07T09:07:06.1161873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1161937Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1162210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1162276Z     self.key(current_states)
2025-09-07T09:07:06.1162282Z 
2025-09-07T09:07:06.1162376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1162559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1162621Z     return mod(**inputs)
2025-09-07T09:07:06.1162895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1162959Z     outputs = self.bert(
2025-09-07T09:07:06.1163201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1163266Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1163506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1163572Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1163792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1163858Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1164096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1164173Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1164408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1164470Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1164709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1164771Z     self_outputs = self.self(
2025-09-07T09:07:06.1165009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1165071Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1165312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1165375Z     self.value(current_states)
2025-09-07T09:07:06.1165378Z 
2025-09-07T09:07:06.1165452Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1165543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1165727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1165785Z     return mod(**inputs)
2025-09-07T09:07:06.1166026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1166082Z     outputs = self.bert(
2025-09-07T09:07:06.1166324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1166423Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1166696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1166761Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1166973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1167040Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1167281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1167356Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1167588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1167651Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1167921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1167991Z     self_outputs = self.self(
2025-09-07T09:07:06.1168225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1168283Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1168558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1168683Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1168686Z 
2025-09-07T09:07:06.1168777Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1168960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1169018Z     return mod(**inputs)
2025-09-07T09:07:06.1169266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1169325Z     outputs = self.bert(
2025-09-07T09:07:06.1169569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1169631Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1169877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1169939Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1170149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1170217Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1170455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1170529Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1170762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1170825Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1171065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1171188Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1171433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1171506Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1171509Z 
2025-09-07T09:07:06.1171595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1171784Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1171845Z     return mod(**inputs)
2025-09-07T09:07:06.1172098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1172220Z     outputs = self.bert(
2025-09-07T09:07:06.1172463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1172532Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1172772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1172836Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1173049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1173116Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1173356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1173429Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1173732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1173801Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1174077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1174186Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1174457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1174536Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1174539Z 
2025-09-07T09:07:06.1174627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1174813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1174871Z     return mod(**inputs)
2025-09-07T09:07:06.1175117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1175176Z     outputs = self.bert(
2025-09-07T09:07:06.1175423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1175491Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1175740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1175805Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1176017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1176082Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1176326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1176397Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1176656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1176722Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1176993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1177103Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1177345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1177486Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1177693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1177758Z     return self.act(input)
2025-09-07T09:07:06.1177761Z 
2025-09-07T09:07:06.1177850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1178075Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1178168Z     return mod(**inputs)
2025-09-07T09:07:06.1178413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1178473Z     outputs = self.bert(
2025-09-07T09:07:06.1178716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1178779Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1179023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1179084Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1179297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1179365Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1179642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1179716Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1179968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1180070Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1180345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1180469Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1180709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1180781Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1180785Z 
2025-09-07T09:07:06.1180884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1181066Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1181125Z     return mod(**inputs)
2025-09-07T09:07:06.1181367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1181424Z     outputs = self.bert(
2025-09-07T09:07:06.1181672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1181734Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1181979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1182039Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1182254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1182322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1182562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1182635Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1182869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1182932Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1183168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1183227Z     self_outputs = self.self(
2025-09-07T09:07:06.1183462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1183520Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1183760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1184040Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1184044Z 
2025-09-07T09:07:06.1184137Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1184319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1184377Z     return mod(**inputs)
2025-09-07T09:07:06.1184624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1184681Z     outputs = self.bert(
2025-09-07T09:07:06.1184927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1184989Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1185264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1185337Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1185552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1185625Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1185899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1185972Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1186210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1186271Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1186511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1186573Z     self_outputs = self.self(
2025-09-07T09:07:06.1186810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1186871Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1187109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1187172Z     self.key(current_states)
2025-09-07T09:07:06.1187175Z 
2025-09-07T09:07:06.1187268Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1187454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1187511Z     return mod(**inputs)
2025-09-07T09:07:06.1187754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1187811Z     outputs = self.bert(
2025-09-07T09:07:06.1188053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1188120Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1188359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1188419Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1188630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1188697Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1188935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1189005Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1189241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1189298Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1189536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1189661Z     self_outputs = self.self(
2025-09-07T09:07:06.1189893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1189953Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1190190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1190254Z     self.value(current_states)
2025-09-07T09:07:06.1190257Z 
2025-09-07T09:07:06.1190332Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1190423Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1190609Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1190665Z     return mod(**inputs)
2025-09-07T09:07:06.1190940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1191004Z     outputs = self.bert(
2025-09-07T09:07:06.1191246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1191311Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1191583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1191646Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1191861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1191928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1192172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1192240Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1192479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1192537Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1192777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1192838Z     self_outputs = self.self(
2025-09-07T09:07:06.1193073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1193134Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1193374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1193495Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1193498Z 
2025-09-07T09:07:06.1193589Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1193773Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1193834Z     return mod(**inputs)
2025-09-07T09:07:06.1194074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1194133Z     outputs = self.bert(
2025-09-07T09:07:06.1194375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1194438Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1194677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1194737Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1194953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1195017Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1195293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1195399Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1195631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1195693Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1195932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1196051Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1196291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1196364Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1196368Z 
2025-09-07T09:07:06.1196458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1196672Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1196733Z     return mod(**inputs)
2025-09-07T09:07:06.1196975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1197032Z     outputs = self.bert(
2025-09-07T09:07:06.1197340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1197406Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1197645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1197706Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1197918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1197986Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1198227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1198304Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1198561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1198628Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1198907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1199019Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1199262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1199331Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1199334Z 
2025-09-07T09:07:06.1199426Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1199612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1199671Z     return mod(**inputs)
2025-09-07T09:07:06.1199918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1199976Z     outputs = self.bert(
2025-09-07T09:07:06.1200223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1200283Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1200525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1200590Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1200801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1200869Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1201149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1201259Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1201514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1201581Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1201856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1201961Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1202204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1202305Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1202543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1202612Z     return self.act(input)
2025-09-07T09:07:06.1202615Z 
2025-09-07T09:07:06.1202704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1202890Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1202949Z     return mod(**inputs)
2025-09-07T09:07:06.1203444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1203505Z     outputs = self.bert(
2025-09-07T09:07:06.1203756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1203827Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1204075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1204149Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1204369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1204442Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1204704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1204785Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1205047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1205115Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1205393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1205523Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1205770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1205845Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1205847Z 
2025-09-07T09:07:06.1205943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1206141Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1206200Z     return mod(**inputs)
2025-09-07T09:07:06.1206448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1206511Z     outputs = self.bert(
2025-09-07T09:07:06.1206755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1206825Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1207067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1207183Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1207455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1207527Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1207780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1207858Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1208102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1208169Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1208413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1208482Z     self_outputs = self.self(
2025-09-07T09:07:06.1208771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1208840Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1209082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1209288Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1209326Z 
2025-09-07T09:07:06.1209431Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1209625Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1209691Z     return mod(**inputs)
2025-09-07T09:07:06.1209940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1210000Z     outputs = self.bert(
2025-09-07T09:07:06.1210251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1210320Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1210565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1210628Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1210851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1210920Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1211162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1211237Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1211471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1211536Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1211777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1211840Z     self_outputs = self.self(
2025-09-07T09:07:06.1212082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1212142Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1212385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1212446Z     self.key(current_states)
2025-09-07T09:07:06.1212450Z 
2025-09-07T09:07:06.1212545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1212742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1212798Z     return mod(**inputs)
2025-09-07T09:07:06.1213048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1213166Z     outputs = self.bert(
2025-09-07T09:07:06.1213448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1213515Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1213758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1213821Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1214034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1214104Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1214343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1214411Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1214687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1214750Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1214991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1215053Z     self_outputs = self.self(
2025-09-07T09:07:06.1215320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1215383Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1215626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1215696Z     self.value(current_states)
2025-09-07T09:07:06.1215699Z 
2025-09-07T09:07:06.1215772Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1215867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1216055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1216112Z     return mod(**inputs)
2025-09-07T09:07:06.1216355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1216412Z     outputs = self.bert(
2025-09-07T09:07:06.1216661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1216724Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1216963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1217028Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1217245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1217320Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1217670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1217745Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1217989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1218050Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1218299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1218360Z     self_outputs = self.self(
2025-09-07T09:07:06.1218603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1218664Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1218903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1219086Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1219123Z 
2025-09-07T09:07:06.1219218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1219408Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1219469Z     return mod(**inputs)
2025-09-07T09:07:06.1219715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1219778Z     outputs = self.bert(
2025-09-07T09:07:06.1220020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1220086Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1220324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1220387Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1220655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1220728Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1220972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1221040Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1221318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1221380Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1221621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1221742Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1221984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1222065Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1222070Z 
2025-09-07T09:07:06.1222160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1222346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1222406Z     return mod(**inputs)
2025-09-07T09:07:06.1222654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1222711Z     outputs = self.bert(
2025-09-07T09:07:06.1222954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1223019Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1223263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1223323Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1223544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1223613Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1223855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1223927Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1224195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1224269Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1224545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1224664Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1224908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1225048Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1225053Z 
2025-09-07T09:07:06.1225146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1225331Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1225393Z     return mod(**inputs)
2025-09-07T09:07:06.1225637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1225698Z     outputs = self.bert(
2025-09-07T09:07:06.1225941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1226002Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1226248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1226365Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1226585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1226654Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1226895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1227011Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1227273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1227344Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1227635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1227741Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1227994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1228101Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1228316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1228376Z     return self.act(input)
2025-09-07T09:07:06.1228380Z 
2025-09-07T09:07:06.1228477Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1228661Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1228716Z     return mod(**inputs)
2025-09-07T09:07:06.1228962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1229017Z     outputs = self.bert(
2025-09-07T09:07:06.1229264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1229330Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1229570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1229634Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1229846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1229918Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1230158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1230230Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1230483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1230547Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1230820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1231014Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1231259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1231329Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1231332Z 
2025-09-07T09:07:06.1231420Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1231605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1231660Z     return mod(**inputs)
2025-09-07T09:07:06.1231905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1231960Z     outputs = self.bert(
2025-09-07T09:07:06.1232237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1232307Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1232548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1232612Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1232860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1232931Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1233169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1233241Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1233481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1233541Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1233788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1233851Z     self_outputs = self.self(
2025-09-07T09:07:06.1234085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1234146Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1234385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1234592Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1234595Z 
2025-09-07T09:07:06.1234686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1234872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1234927Z     return mod(**inputs)
2025-09-07T09:07:06.1235174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1235233Z     outputs = self.bert(
2025-09-07T09:07:06.1235474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1235538Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1235777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1235837Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1236049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1236116Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1236360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1236429Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1236715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1236811Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1237050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1237115Z     self_outputs = self.self(
2025-09-07T09:07:06.1237350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1237410Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1237649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1237709Z     self.key(current_states)
2025-09-07T09:07:06.1237712Z 
2025-09-07T09:07:06.1237803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1238020Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1238082Z     return mod(**inputs)
2025-09-07T09:07:06.1238325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1238383Z     outputs = self.bert(
2025-09-07T09:07:06.1238659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1238724Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1238966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1239026Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1239237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1239306Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1239547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1239620Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1239851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1239910Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1240150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1240210Z     self_outputs = self.self(
2025-09-07T09:07:06.1240445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1240504Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1240743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1240807Z     self.value(current_states)
2025-09-07T09:07:06.1240811Z 
2025-09-07T09:07:06.1240884Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1240973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1241154Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1241211Z     return mod(**inputs)
2025-09-07T09:07:06.1241455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1241511Z     outputs = self.bert(
2025-09-07T09:07:06.1241754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1241815Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1242056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1242114Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1242366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1242466Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1242706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1242778Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1243011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1243070Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1243308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1243368Z     self_outputs = self.self(
2025-09-07T09:07:06.1243604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1243713Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1243962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1244084Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1244087Z 
2025-09-07T09:07:06.1244178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1244400Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1244458Z     return mod(**inputs)
2025-09-07T09:07:06.1244705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1244763Z     outputs = self.bert(
2025-09-07T09:07:06.1245010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1245073Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1245315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1245381Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1245597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1245668Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1245909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1245977Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1246215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1246276Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1246525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1246648Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1246888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1246963Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1246967Z 
2025-09-07T09:07:06.1247056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1247244Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1247298Z     return mod(**inputs)
2025-09-07T09:07:06.1247543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1247598Z     outputs = self.bert(
2025-09-07T09:07:06.1247838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1247944Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1248187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1248300Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1248513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1248578Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1248819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1248891Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1249153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1249221Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1249531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1249648Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1249893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1249969Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1249973Z 
2025-09-07T09:07:06.1250091Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1250281Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1250337Z     return mod(**inputs)
2025-09-07T09:07:06.1250582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1250642Z     outputs = self.bert(
2025-09-07T09:07:06.1250884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1250951Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1251190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1251250Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1251463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1251529Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1251769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1251839Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1252098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1252162Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1252434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1252543Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1252782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1252884Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1253094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1253154Z     return self.act(input)
2025-09-07T09:07:06.1253157Z 
2025-09-07T09:07:06.1253251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1253434Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1253496Z     return mod(**inputs)
2025-09-07T09:07:06.1253744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1253836Z     outputs = self.bert(
2025-09-07T09:07:06.1254119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1254187Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1254435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1254495Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1254713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1254780Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1255025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1255103Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1255391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1255467Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1255742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1255868Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1256147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1256222Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1256225Z 
2025-09-07T09:07:06.1256321Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1256507Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1256570Z     return mod(**inputs)
2025-09-07T09:07:06.1256820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1256879Z     outputs = self.bert(
2025-09-07T09:07:06.1257124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1257187Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1257472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1257533Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1257742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1257812Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1258051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1258122Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1258362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1258423Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1258664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1258727Z     self_outputs = self.self(
2025-09-07T09:07:06.1258965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1259023Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1259264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1259466Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1259469Z 
2025-09-07T09:07:06.1259558Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1259788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1259877Z     return mod(**inputs)
2025-09-07T09:07:06.1260123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1260180Z     outputs = self.bert(
2025-09-07T09:07:06.1260424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1260491Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1260730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1260792Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1261003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1261071Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1261345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1261418Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1261653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1261743Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1261988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1262049Z     self_outputs = self.self(
2025-09-07T09:07:06.1262285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1262346Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1262584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1262649Z     self.key(current_states)
2025-09-07T09:07:06.1262653Z 
2025-09-07T09:07:06.1262741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1262922Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1262979Z     return mod(**inputs)
2025-09-07T09:07:06.1263223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1263281Z     outputs = self.bert(
2025-09-07T09:07:06.1263522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1263586Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1263824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1263885Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1264101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1264169Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1264408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1264475Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1264708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1264767Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1265005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1265067Z     self_outputs = self.self(
2025-09-07T09:07:06.1265303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1265398Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1265641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1265742Z     self.value(current_states)
2025-09-07T09:07:06.1265745Z 
2025-09-07T09:07:06.1265822Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1265911Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1266094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1266153Z     return mod(**inputs)
2025-09-07T09:07:06.1266398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1266460Z     outputs = self.bert(
2025-09-07T09:07:06.1266703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1266769Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1267043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1267108Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1267320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1267385Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1267659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1267730Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1267966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1268041Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1268280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1268349Z     self_outputs = self.self(
2025-09-07T09:07:06.1268586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1268647Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1268893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1269015Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1269019Z 
2025-09-07T09:07:06.1269111Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1269293Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1269351Z     return mod(**inputs)
2025-09-07T09:07:06.1269593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1269650Z     outputs = self.bert(
2025-09-07T09:07:06.1269894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1269956Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1270197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1270256Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1270467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1270534Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1270772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1270842Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1271076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1271189Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1271467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1271584Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1271831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1271904Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1271908Z 
2025-09-07T09:07:06.1272000Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1272182Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1272237Z     return mod(**inputs)
2025-09-07T09:07:06.1272483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1272577Z     outputs = self.bert(
2025-09-07T09:07:06.1272832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1272895Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1273137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1273236Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1273453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1273525Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1273766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1273843Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1274106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1274178Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1274460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1274573Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1274823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1274895Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1274899Z 
2025-09-07T09:07:06.1274991Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1275186Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1275246Z     return mod(**inputs)
2025-09-07T09:07:06.1275505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1275568Z     outputs = self.bert(
2025-09-07T09:07:06.1275819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1275888Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1276127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1276198Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1276418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1276492Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1276730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1276802Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1277065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1277202Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1277481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1277589Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1277832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1277941Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1278149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1278219Z     return self.act(input)
2025-09-07T09:07:06.1278223Z 
2025-09-07T09:07:06.1278316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1278542Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1278602Z     return mod(**inputs)
2025-09-07T09:07:06.1278849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1278913Z     outputs = self.bert(
2025-09-07T09:07:06.1279194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1279266Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1279505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1279566Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1279786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1279854Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1280098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1280170Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1280422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1280489Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1280760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1280888Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1281129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1281202Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1281205Z 
2025-09-07T09:07:06.1281293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1281479Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1281543Z     return mod(**inputs)
2025-09-07T09:07:06.1281787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1288757Z     outputs = self.bert(
2025-09-07T09:07:06.1289083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1289161Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1289445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1289515Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1289765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1289843Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1290197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1290313Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1290566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1290630Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1290876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1290946Z     self_outputs = self.self(
2025-09-07T09:07:06.1291184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1291244Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1291484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1291729Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1291735Z 
2025-09-07T09:07:06.1291841Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1292044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1292107Z     return mod(**inputs)
2025-09-07T09:07:06.1292587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1292649Z     outputs = self.bert(
2025-09-07T09:07:06.1292896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1292965Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1293211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1293278Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1293502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1293578Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1293826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1293899Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1294149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1294218Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1294477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1294542Z     self_outputs = self.self(
2025-09-07T09:07:06.1294779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1294841Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1295088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1295152Z     self.key(current_states)
2025-09-07T09:07:06.1295156Z 
2025-09-07T09:07:06.1295263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1295461Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1295520Z     return mod(**inputs)
2025-09-07T09:07:06.1295770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1295830Z     outputs = self.bert(
2025-09-07T09:07:06.1296075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1296140Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1296474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1296593Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1296809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1296881Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1297125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1297199Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1297500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1297561Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1297803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1297901Z     self_outputs = self.self(
2025-09-07T09:07:06.1298143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1298203Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1298444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1298542Z     self.value(current_states)
2025-09-07T09:07:06.1298546Z 
2025-09-07T09:07:06.1298623Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1298719Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1298905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1298963Z     return mod(**inputs)
2025-09-07T09:07:06.1299211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1299271Z     outputs = self.bert(
2025-09-07T09:07:06.1299516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1299580Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1299825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1299890Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1300106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1300180Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1300421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1300495Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1300732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1300794Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1301038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1301098Z     self_outputs = self.self(
2025-09-07T09:07:06.1301336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1301395Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1301635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1301771Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1301774Z 
2025-09-07T09:07:06.1301870Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1302067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1302162Z     return mod(**inputs)
2025-09-07T09:07:06.1302446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1302506Z     outputs = self.bert(
2025-09-07T09:07:06.1302754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1302825Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1303258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1303328Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1303544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1303615Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1303946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1304025Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1304272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1304335Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1304638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1304768Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1305008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1305090Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1305093Z 
2025-09-07T09:07:06.1305187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1305385Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1305448Z     return mod(**inputs)
2025-09-07T09:07:06.1305699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1305761Z     outputs = self.bert(
2025-09-07T09:07:06.1306006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1306079Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1306320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1306385Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1306607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1306677Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1306922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1307001Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1307268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1307340Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1307622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1307744Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1307989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1308066Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1308070Z 
2025-09-07T09:07:06.1308162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1308355Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1308511Z     return mod(**inputs)
2025-09-07T09:07:06.1308756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1308822Z     outputs = self.bert(
2025-09-07T09:07:06.1309068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1309137Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1309379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1309442Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1309659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1309729Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1310004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1310082Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1310339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1310410Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1310723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1310838Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1311079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1311182Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1311396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1311462Z     return self.act(input)
2025-09-07T09:07:06.1311466Z 
2025-09-07T09:07:06.1311563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1311754Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1311822Z     return mod(**inputs)
2025-09-07T09:07:06.1312070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1312127Z     outputs = self.bert(
2025-09-07T09:07:06.1312378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1312443Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1312687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1312747Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1312963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1313036Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1313275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1313350Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1313606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1313673Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1313946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1314071Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1314321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1314433Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1314465Z 
2025-09-07T09:07:06.1314563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1314750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1314810Z     return mod(**inputs)
2025-09-07T09:07:06.1315062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1315119Z     outputs = self.bert(
2025-09-07T09:07:06.1315365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1315430Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1315668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1315734Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1315983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1316059Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1316301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1316410Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1316663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1316727Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1316983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1317048Z     self_outputs = self.self(
2025-09-07T09:07:06.1317292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1317357Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1317602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward
2025-09-07T09:07:06.1317811Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:07:06.1317815Z 
2025-09-07T09:07:06.1317909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1318104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1318160Z     return mod(**inputs)
2025-09-07T09:07:06.1318409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1318467Z     outputs = self.bert(
2025-09-07T09:07:06.1318712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1318783Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1319025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1319089Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1319300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1319369Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1319613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1319686Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1319931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1319995Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1320234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1320393Z     self_outputs = self.self(
2025-09-07T09:07:06.1320629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1320694Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1320936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward
2025-09-07T09:07:06.1320998Z     self.key(current_states)
2025-09-07T09:07:06.1321003Z 
2025-09-07T09:07:06.1321092Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1321275Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1321332Z     return mod(**inputs)
2025-09-07T09:07:06.1321576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1321635Z     outputs = self.bert(
2025-09-07T09:07:06.1321928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1321995Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1322236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1322329Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1322549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1322617Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1322857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1322928Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1323162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1323224Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1323464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1323524Z     self_outputs = self.self(
2025-09-07T09:07:06.1323761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1323820Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1324060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward
2025-09-07T09:07:06.1324123Z     self.value(current_states)
2025-09-07T09:07:06.1324126Z 
2025-09-07T09:07:06.1324201Z cudagraph partition due to non gpu ops
2025-09-07T09:07:06.1324290Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1324474Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1324539Z     return mod(**inputs)
2025-09-07T09:07:06.1324784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1324843Z     outputs = self.bert(
2025-09-07T09:07:06.1325084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1325147Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1325390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1325451Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1325665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1325730Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1325970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1326117Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1326349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1326409Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1326651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward
2025-09-07T09:07:06.1326715Z     self_outputs = self.self(
2025-09-07T09:07:06.1326946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1327005Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1327245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward
2025-09-07T09:07:06.1327369Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:07:06.1327374Z 
2025-09-07T09:07:06.1327503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1327691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1327748Z     return mod(**inputs)
2025-09-07T09:07:06.1327994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1328083Z     outputs = self.bert(
2025-09-07T09:07:06.1328330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1328394Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1328632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1328695Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1328909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1328978Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1329216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward
2025-09-07T09:07:06.1329287Z     self_attention_outputs = self.attention(
2025-09-07T09:07:06.1329521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:07:06.1329580Z     return func(*args, **kwargs)
2025-09-07T09:07:06.1329823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward
2025-09-07T09:07:06.1329948Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:06.1330193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward
2025-09-07T09:07:06.1330271Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1330277Z 
2025-09-07T09:07:06.1330367Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1330557Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1330616Z     return mod(**inputs)
2025-09-07T09:07:06.1330869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1330927Z     outputs = self.bert(
2025-09-07T09:07:06.1331173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1331239Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1331479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1331542Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1331754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1331888Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1332129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1332205Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1332475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1332543Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1332826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1332938Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1333184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward
2025-09-07T09:07:06.1333293Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1333298Z 
2025-09-07T09:07:06.1333389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1333583Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1333642Z     return mod(**inputs)
2025-09-07T09:07:06.1333925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1333984Z     outputs = self.bert(
2025-09-07T09:07:06.1334226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1334295Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1334537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1334608Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1334825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1334896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1335142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1335224Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1335484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1335555Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1335825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk
2025-09-07T09:07:06.1335931Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:06.1336175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward
2025-09-07T09:07:06.1336280Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:06.1336491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:07:06.1336554Z     return self.act(input)
2025-09-07T09:07:06.1336557Z 
2025-09-07T09:07:06.1336653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1336840Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1336896Z     return mod(**inputs)
2025-09-07T09:07:06.1337141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward
2025-09-07T09:07:06.1337199Z     outputs = self.bert(
2025-09-07T09:07:06.1337503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward
2025-09-07T09:07:06.1337570Z     encoder_outputs = self.encoder(
2025-09-07T09:07:06.1337850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward
2025-09-07T09:07:06.1337950Z     layer_outputs = layer_module(
2025-09-07T09:07:06.1338164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:07:06.1338236Z     return super().__call__(*args, **kwargs)
2025-09-07T09:07:06.1338479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward
2025-09-07T09:07:06.1338553Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:06.1338814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:06.1338878Z     return forward_fn(*input_tensors)
2025-09-07T09:07:06.1339152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk
2025-09-07T09:07:06.1339310Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:06.1339559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward
2025-09-07T09:07:06.1339629Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1339632Z 
2025-09-07T09:07:06.1339769Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1339958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1340015Z     return mod(**inputs)
2025-09-07T09:07:06.1340266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T09:07:06.1340352Z     prediction_scores = self.cls(sequence_output)
2025-09-07T09:07:06.1340593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T09:07:06.1340704Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T09:07:06.1340946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward
2025-09-07T09:07:06.1341029Z     hidden_states = self.transform(hidden_states)
2025-09-07T09:07:06.1341272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward
2025-09-07T09:07:06.1341343Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:06.1341347Z 
2025-09-07T09:07:06.1341436Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:06.1341618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:07:06.1341676Z     return mod(**inputs)
2025-09-07T09:07:06.1341920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward
2025-09-07T09:07:06.1342003Z     prediction_scores = self.cls(sequence_output)
2025-09-07T09:07:06.1342243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward
2025-09-07T09:07:06.1342338Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T09:07:06.1342581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward
2025-09-07T09:07:06.1342664Z     hidden_states = self.decoder(hidden_states)
2025-09-07T09:07:06.1342667Z 
2025-09-07T09:07:16.5001435Z 
2025-09-07T09:07:16.7249900Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:07:16.9635378Z running benchmark:   3% 1/30 [00:00<00:06,  4.45it/s]
2025-09-07T09:07:17.2005311Z running benchmark:   7% 2/30 [00:00<00:06,  4.29it/s]
2025-09-07T09:07:17.4309586Z running benchmark:  10% 3/30 [00:00<00:06,  4.26it/s]
2025-09-07T09:07:17.6616145Z running benchmark:  13% 4/30 [00:00<00:06,  4.29it/s]
2025-09-07T09:07:17.8966890Z running benchmark:  17% 5/30 [00:01<00:05,  4.31it/s]
2025-09-07T09:07:18.1254315Z running benchmark:  20% 6/30 [00:01<00:05,  4.29it/s]
2025-09-07T09:07:18.3600266Z running benchmark:  23% 7/30 [00:01<00:05,  4.32it/s]
2025-09-07T09:07:18.5914649Z running benchmark:  27% 8/30 [00:01<00:05,  4.30it/s]
2025-09-07T09:07:18.8299122Z running benchmark:  30% 9/30 [00:02<00:04,  4.31it/s]
2025-09-07T09:07:19.0577002Z running benchmark:  33% 10/30 [00:02<00:04,  4.27it/s]
2025-09-07T09:07:19.2952621Z running benchmark:  37% 11/30 [00:02<00:04,  4.31it/s]
2025-09-07T09:07:19.5256627Z running benchmark:  40% 12/30 [00:02<00:04,  4.28it/s]
2025-09-07T09:07:19.7618375Z running benchmark:  43% 13/30 [00:03<00:03,  4.30it/s]
2025-09-07T09:07:19.9926152Z running benchmark:  47% 14/30 [00:03<00:03,  4.28it/s]
2025-09-07T09:07:20.2969193Z running benchmark:  50% 15/30 [00:03<00:03,  4.29it/s]
2025-09-07T09:07:20.5276105Z running benchmark:  53% 16/30 [00:03<00:03,  3.93it/s]
2025-09-07T09:07:20.7649996Z running benchmark:  57% 17/30 [00:04<00:03,  4.04it/s]
2025-09-07T09:07:20.9948874Z running benchmark:  60% 18/30 [00:04<00:02,  4.09it/s]
2025-09-07T09:07:21.2269082Z running benchmark:  63% 19/30 [00:04<00:02,  4.17it/s]
2025-09-07T09:07:21.4567455Z running benchmark:  67% 20/30 [00:04<00:02,  4.21it/s]
2025-09-07T09:07:21.6918400Z running benchmark:  70% 21/30 [00:04<00:02,  4.25it/s]
2025-09-07T09:07:21.9205858Z running benchmark:  73% 22/30 [00:05<00:01,  4.25it/s]
2025-09-07T09:07:22.1582253Z running benchmark:  77% 23/30 [00:05<00:01,  4.29it/s]
2025-09-07T09:07:22.3896668Z running benchmark:  80% 24/30 [00:05<00:01,  4.26it/s]
2025-09-07T09:07:22.6253689Z running benchmark:  83% 25/30 [00:05<00:01,  4.28it/s]
2025-09-07T09:07:22.8564009Z running benchmark:  87% 26/30 [00:06<00:00,  4.27it/s]
2025-09-07T09:07:23.0930394Z running benchmark:  90% 27/30 [00:06<00:00,  4.29it/s]
2025-09-07T09:07:23.3297144Z running benchmark:  93% 28/30 [00:06<00:00,  4.27it/s]
2025-09-07T09:07:23.5639971Z running benchmark:  97% 29/30 [00:06<00:00,  4.25it/s]
2025-09-07T09:07:23.5640407Z running benchmark: 100% 30/30 [00:07<00:00,  4.26it/s]
2025-09-07T09:07:23.5640738Z running benchmark: 100% 30/30 [00:07<00:00,  4.25it/s]
2025-09-07T09:07:23.7024178Z 1.184x
2025-09-07T09:07:23.7028981Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:07:26.5858295Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:07:26.5859198Z   import pynvml  # type: ignore[import]
2025-09-07T09:07:28.6051302Z 
2025-09-07T09:07:29.9999568Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T09:07:30.0000908Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T09:07:30.0001740Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T09:07:30.0002392Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T09:07:30.0003942Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T09:07:30.0005696Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T09:07:30.0006490Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T09:07:30.0007116Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T09:07:31.6715783Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T09:07:31.6716427Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T09:07:32.2232006Z 
2025-09-07T09:07:32.2233083Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:07:32.2444646Z cpu  eval  hf_BigBird                         
2025-09-07T09:07:36.7819264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:36.7819798Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1897, in forward
2025-09-07T09:07:36.7820752Z     attention_mask = torch.ones(((batch_size, seq_length + past_key_values_length)), device=device)
2025-09-07T09:07:36.7821011Z 
2025-09-07T09:07:41.8538955Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:41.8539625Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2090, in torch_dynamo_resume_in__pad_to_block_size_at_2085
2025-09-07T09:07:41.8540254Z     input_ids = nn.functional.pad(input_ids, (0, padding_len), value=pad_token_id)
2025-09-07T09:07:41.8540723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:07:41.8541101Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:07:41.8541260Z 
2025-09-07T09:07:41.8541367Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:41.8541906Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2103, in torch_dynamo_resume_in__pad_to_block_size_at_2085
2025-09-07T09:07:41.8542415Z     attention_mask = nn.functional.pad(
2025-09-07T09:07:41.8542748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:07:41.8543093Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:07:41.8543238Z 
2025-09-07T09:07:41.8543332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:41.8543844Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2106, in torch_dynamo_resume_in__pad_to_block_size_at_2085
2025-09-07T09:07:41.8544481Z     token_type_ids = nn.functional.pad(token_type_ids, (0, padding_len), value=0)  # pad with token_type_id = 0
2025-09-07T09:07:41.8544931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:07:41.8545273Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:07:41.8545419Z 
2025-09-07T09:07:42.7859713Z cudagraph partition due to non gpu ops
2025-09-07T09:07:42.7860005Z cudagraph partition due to non gpu ops
2025-09-07T09:07:42.7860228Z cudagraph partition due to non gpu ops
2025-09-07T09:07:42.7860466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:42.7861029Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1947, in torch_dynamo_resume_in_forward_at_1935
2025-09-07T09:07:42.7864914Z     blocked_encoder_mask, band_mask, from_mask, to_mask = self.create_masks_for_block_sparse_attn(
2025-09-07T09:07:42.7865906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2060, in create_masks_for_block_sparse_attn
2025-09-07T09:07:42.7866575Z     band_mask = create_band_mask_from_inputs(blocked_encoder_mask, blocked_encoder_mask)
2025-09-07T09:07:42.7867130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2052, in create_band_mask_from_inputs
2025-09-07T09:07:42.7867595Z     exp_blocked_to_pad = torch.cat(
2025-09-07T09:07:42.7867729Z 
2025-09-07T09:07:42.7867835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:42.7868338Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1947, in torch_dynamo_resume_in_forward_at_1935
2025-09-07T09:07:42.7868916Z     blocked_encoder_mask, band_mask, from_mask, to_mask = self.create_masks_for_block_sparse_attn(
2025-09-07T09:07:42.7870790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2060, in create_masks_for_block_sparse_attn
2025-09-07T09:07:42.7871345Z     band_mask = create_band_mask_from_inputs(blocked_encoder_mask, blocked_encoder_mask)
2025-09-07T09:07:42.7871962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2055, in create_band_mask_from_inputs
2025-09-07T09:07:42.7872508Z     band_mask = torch.einsum("blq,blk->blqk", from_blocked_mask[:, 2:-2], exp_blocked_to_pad)
2025-09-07T09:07:42.7872727Z 
2025-09-07T09:07:44.1042150Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.1042716Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1362, in forward
2025-09-07T09:07:44.1043164Z     band_mask = band_mask.to(hidden_states.dtype)
2025-09-07T09:07:44.1043310Z 
2025-09-07T09:07:44.1043417Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.1043902Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1364, in forward
2025-09-07T09:07:44.1044330Z     from_mask = from_mask.to(hidden_states.dtype)
2025-09-07T09:07:44.1044473Z 
2025-09-07T09:07:44.1044582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.1045020Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1366, in forward
2025-09-07T09:07:44.1045425Z     to_mask = to_mask.to(hidden_states.dtype)
2025-09-07T09:07:44.1045565Z 
2025-09-07T09:07:44.8390165Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.8390717Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 440, in forward
2025-09-07T09:07:44.8391149Z     self.query(hidden_states)
2025-09-07T09:07:44.8391268Z 
2025-09-07T09:07:44.8391406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.8391878Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 445, in forward
2025-09-07T09:07:44.8392288Z     self.key(hidden_states)
2025-09-07T09:07:44.8392399Z 
2025-09-07T09:07:44.8392508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.8392946Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 450, in forward
2025-09-07T09:07:44.8393340Z     self.value(hidden_states)
2025-09-07T09:07:44.8393462Z 
2025-09-07T09:07:44.9958512Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:44.9959120Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1139, in _bigbird_block_rand_mask_with_head
2025-09-07T09:07:44.9959656Z     plan_block_length = np.array(plan_from_length) // from_block_size
2025-09-07T09:07:44.9960187Z 
2025-09-07T09:07:47.6163821Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6164811Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 602, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6165471Z     first_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 0], key_layer, ndim=4)
2025-09-07T09:07:47.6166035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T09:07:47.6166511Z     return torch.bmm(
2025-09-07T09:07:47.6166613Z 
2025-09-07T09:07:47.6166696Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6166911Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6167102Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6167298Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6167485Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6167790Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6167988Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6168182Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6168375Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6168559Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6168752Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6169018Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6169219Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6169405Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6169596Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6169787Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6170008Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6170552Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 647, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6171147Z     to_mask.new_ones([bsz, 1, 1, n_rand_blocks * to_block_size]),
2025-09-07T09:07:47.6171327Z 
2025-09-07T09:07:47.6171428Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6171985Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 653, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6172555Z     rand_mask.new_ones([bsz, n_heads, from_block_size, 4 * to_block_size]),
2025-09-07T09:07:47.6172742Z 
2025-09-07T09:07:47.6172819Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6173018Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6173236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6173762Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 698, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6174281Z     first_band_product = torch.einsum(
2025-09-07T09:07:47.6174414Z 
2025-09-07T09:07:47.6174485Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6174678Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6174873Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6175124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6175666Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 704, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6176173Z     last_band_product = torch.einsum(
2025-09-07T09:07:47.6176300Z 
2025-09-07T09:07:47.6176384Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6176580Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6176781Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6176976Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6177172Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6177429Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6177704Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6177931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6178470Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 781, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6179022Z     to_mask.new_ones([bsz, 1, 1, n_rand_blocks * to_block_size]),
2025-09-07T09:07:47.6179189Z 
2025-09-07T09:07:47.6179289Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6179824Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 787, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6180393Z     rand_mask.new_ones([bsz, n_heads, from_block_size, 4 * to_block_size]),
2025-09-07T09:07:47.6180579Z 
2025-09-07T09:07:47.6180704Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6180913Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6181131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6181656Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 807, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6183098Z     last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -1], key_layer, ndim=4)
2025-09-07T09:07:47.6183661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T09:07:47.6184110Z     return torch.bmm(
2025-09-07T09:07:47.6184211Z 
2025-09-07T09:07:47.6184311Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6184879Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 611, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6185482Z     first_context_layer = self.torch_bmm_nd(first_attn_weights, value_layer, ndim=4)
2025-09-07T09:07:47.6185987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T09:07:47.6186519Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T09:07:47.6186754Z 
2025-09-07T09:07:47.6186852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6187390Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 574, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6187904Z     rand_attn = np.stack(rand_attn, axis=0)
2025-09-07T09:07:47.6188034Z 
2025-09-07T09:07:47.6188134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6188666Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 575, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6189245Z     rand_attn = torch.tensor(rand_attn, device=query_layer.device, dtype=torch.long)
2025-09-07T09:07:47.6189465Z 
2025-09-07T09:07:47.6189561Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6190087Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 620, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6190591Z     second_key_mat = torch.cat(
2025-09-07T09:07:47.6190703Z 
2025-09-07T09:07:47.6190800Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6191316Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 642, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6192030Z     second_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, 1], second_key_mat, ndim=4)
2025-09-07T09:07:47.6192622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T09:07:47.6193048Z     return torch.bmm(
2025-09-07T09:07:47.6193155Z 
2025-09-07T09:07:47.6193267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6193806Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 643, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6194313Z     second_seq_pad = torch.cat(
2025-09-07T09:07:47.6194439Z 
2025-09-07T09:07:47.6194539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6195122Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 651, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6195642Z     second_rand_pad = torch.cat(
2025-09-07T09:07:47.6195764Z 
2025-09-07T09:07:47.6195865Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6196386Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 630, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6196917Z     second_value_mat = torch.cat(
2025-09-07T09:07:47.6197048Z 
2025-09-07T09:07:47.6197141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6197667Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 665, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6198271Z     second_context_layer = self.torch_bmm_nd(second_attn_weights, second_value_mat, ndim=4)
2025-09-07T09:07:47.6198778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T09:07:47.6199310Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T09:07:47.6199552Z 
2025-09-07T09:07:47.6199646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6200177Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 676, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6200683Z     exp_blocked_key_matrix = torch.cat(
2025-09-07T09:07:47.6200812Z 
2025-09-07T09:07:47.6200909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6201425Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 687, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6202054Z     inner_band_product = self.torch_bmm_nd_transpose(middle_query_matrix, exp_blocked_key_matrix, ndim=5)
2025-09-07T09:07:47.6202617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T09:07:47.6203230Z     return torch.bmm(
2025-09-07T09:07:47.6203321Z 
2025-09-07T09:07:47.6203419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6203938Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 693, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6204568Z     rand_band_product = self.torch_bmm_nd_transpose(middle_query_matrix, gathered_key[:, :, 1:-1], ndim=5)
2025-09-07T09:07:47.6205123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T09:07:47.6205543Z     return torch.bmm(
2025-09-07T09:07:47.6205695Z 
2025-09-07T09:07:47.6205794Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6206362Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 711, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6206955Z     first_band_product += (1.0 - to_mask[:, :, :, :to_block_size].unsqueeze(3)) * attn_mask_penalty
2025-09-07T09:07:47.6207173Z 
2025-09-07T09:07:47.6207268Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6207796Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 710, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6208346Z     inner_band_product += (1.0 - band_mask) * attn_mask_penalty
2025-09-07T09:07:47.6208512Z 
2025-09-07T09:07:47.6208610Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6209168Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 713, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6209740Z     rand_band_product += (1.0 - rand_mask[:, :, 1:-1]) * attn_mask_penalty
2025-09-07T09:07:47.6209929Z 
2025-09-07T09:07:47.6210029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6210620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 712, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6211220Z     last_band_product += (1.0 - to_mask[:, :, :, -to_block_size:].unsqueeze(3)) * attn_mask_penalty
2025-09-07T09:07:47.6211431Z 
2025-09-07T09:07:47.6211537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6212071Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 716, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6212576Z     band_product = torch.cat(
2025-09-07T09:07:47.6212693Z 
2025-09-07T09:07:47.6212797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6213325Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 679, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6213833Z     exp_blocked_value_matrix = torch.cat(
2025-09-07T09:07:47.6213973Z 
2025-09-07T09:07:47.6214063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6214588Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 727, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6215100Z     context_layer = self.torch_bmm_nd(
2025-09-07T09:07:47.6215511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T09:07:47.6216039Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T09:07:47.6216284Z 
2025-09-07T09:07:47.6216375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6216903Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 734, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6217426Z     context_layer += self.torch_bmm_nd(
2025-09-07T09:07:47.6217872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T09:07:47.6218391Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T09:07:47.6218623Z 
2025-09-07T09:07:47.6218716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6219241Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 740, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6219825Z     context_layer += torch.einsum(
2025-09-07T09:07:47.6219948Z 
2025-09-07T09:07:47.6220047Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6220571Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 743, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6221070Z     context_layer += torch.einsum(
2025-09-07T09:07:47.6221191Z 
2025-09-07T09:07:47.6221282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6221805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 754, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6222307Z     second_last_key_mat = torch.cat(
2025-09-07T09:07:47.6222432Z 
2025-09-07T09:07:47.6222565Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6223081Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 776, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6223726Z     second_last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -2], second_last_key_mat, ndim=4)
2025-09-07T09:07:47.6224331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 493, in torch_bmm_nd_transpose
2025-09-07T09:07:47.6224761Z     return torch.bmm(
2025-09-07T09:07:47.6224849Z 
2025-09-07T09:07:47.6224944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6225462Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 777, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6225970Z     second_last_seq_pad = torch.cat(
2025-09-07T09:07:47.6226097Z 
2025-09-07T09:07:47.6226191Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6226711Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 785, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6227214Z     second_last_rand_pad = torch.cat(
2025-09-07T09:07:47.6227338Z 
2025-09-07T09:07:47.6227440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6227958Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 764, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6228461Z     second_last_value_mat = torch.cat(
2025-09-07T09:07:47.6228592Z 
2025-09-07T09:07:47.6228683Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6229207Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 799, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6229835Z     second_last_context_layer = self.torch_bmm_nd(second_last_attn_weights, second_last_value_mat, ndim=4)
2025-09-07T09:07:47.6230360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T09:07:47.6230879Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T09:07:47.6231116Z 
2025-09-07T09:07:47.6231206Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6231734Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 813, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6232311Z     last_context_layer = self.torch_bmm_nd(last_attn_weights, value_layer, ndim=4)
2025-09-07T09:07:47.6232900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 485, in torch_bmm_nd
2025-09-07T09:07:47.6233452Z     return torch.bmm(inp_1.reshape((-1,) + inp_1.shape[-2:]), inp_2.reshape((-1,) + inp_2.shape[-2:])).view(
2025-09-07T09:07:47.6233686Z 
2025-09-07T09:07:47.6233761Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6233964Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6234199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6234713Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 743, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6235216Z     context_layer += torch.einsum(
2025-09-07T09:07:47.6235350Z 
2025-09-07T09:07:47.6235420Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6235630Z cudagraph partition due to non gpu ops
2025-09-07T09:07:47.6235883Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6236403Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 817, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6236911Z     context_layer = torch.cat(
2025-09-07T09:07:47.6237026Z 
2025-09-07T09:07:47.6237116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:47.6237686Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 821, in torch_dynamo_resume_in_bigbird_block_sparse_attention_at_564
2025-09-07T09:07:47.6238280Z     context_layer = context_layer.view((bsz, n_heads, from_seq_len, -1)) * from_mask
2025-09-07T09:07:47.6238492Z 
2025-09-07T09:07:56.4275723Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:56.4276365Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 478, in torch_dynamo_resume_in_forward_at_455
2025-09-07T09:07:56.4276967Z     context_layer = context_layer.contiguous().view(batch_size, from_seq_length, -1)
2025-09-07T09:07:56.4277196Z 
2025-09-07T09:07:57.1445857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:57.1446477Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1385, in torch_dynamo_resume_in_forward_at_1381
2025-09-07T09:07:57.1447019Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:07:57.1447506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1295, in forward
2025-09-07T09:07:57.1447934Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:57.1448074Z 
2025-09-07T09:07:58.1111306Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:58.1111904Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1505, in torch_dynamo_resume_in_forward_at_1468
2025-09-07T09:07:58.1112408Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:58.1112828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:58.1113241Z     return forward_fn(*input_tensors)
2025-09-07T09:07:58.1113711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1512, in feed_forward_chunk
2025-09-07T09:07:58.1114197Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:58.1114641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward
2025-09-07T09:07:58.1115058Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:58.1115200Z 
2025-09-07T09:07:58.1115302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:58.1116108Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1505, in torch_dynamo_resume_in_forward_at_1468
2025-09-07T09:07:58.1116693Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:58.1117093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:58.1117490Z     return forward_fn(*input_tensors)
2025-09-07T09:07:58.1117913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1512, in feed_forward_chunk
2025-09-07T09:07:58.1118384Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:07:58.1118822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1402, in forward
2025-09-07T09:07:58.1119252Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:07:58.1119719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:07:58.1120191Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:07:58.1120431Z 
2025-09-07T09:07:58.1120542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:07:58.1121109Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1505, in torch_dynamo_resume_in_forward_at_1468
2025-09-07T09:07:58.1121585Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:07:58.1121988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:07:58.1122383Z     return forward_fn(*input_tensors)
2025-09-07T09:07:58.1122799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1513, in feed_forward_chunk
2025-09-07T09:07:58.1123293Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:07:58.1123750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1415, in forward
2025-09-07T09:07:58.1124162Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:07:58.1124307Z 
2025-09-07T09:08:05.3333183Z W0907 09:08:05.332554 168802 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] torch._dynamo hit config.recompile_limit (8)
2025-09-07T09:08:05.3334104Z W0907 09:08:05.332554 168802 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    function: 'forward' (/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py:417)
2025-09-07T09:08:05.3335870Z W0907 09:08:05.332554 168802 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    last reason: 10/7: self.seed == 7                                           # context_layer, attention_probs = self.bigbird_block_sparse_attention(  # transformers/models/big_bird/modeling_big_bird.py:455 in forward (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
2025-09-07T09:08:05.3337470Z W0907 09:08:05.332554 168802 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-09-07T09:08:05.3338355Z W0907 09:08:05.332554 168802 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2025-09-07T09:08:05.9734315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:05.9734948Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2010, in torch_dynamo_resume_in_forward_at_1991
2025-09-07T09:08:05.9735990Z     pooler_output = self.activation(self.pooler(sequence_output[:, 0, :])) if (self.pooler is not None) else None
2025-09-07T09:08:05.9736364Z 
2025-09-07T09:08:05.9736481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:05.9736983Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2010, in torch_dynamo_resume_in_forward_at_1991
2025-09-07T09:08:05.9737663Z     pooler_output = self.activation(self.pooler(sequence_output[:, 0, :])) if (self.pooler is not None) else None
2025-09-07T09:08:05.9737937Z 
2025-09-07T09:08:06.1779177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:06.1779788Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2318, in torch_dynamo_resume_in_forward_at_2303
2025-09-07T09:08:06.1780301Z     prediction_scores = self.cls(sequence_output)
2025-09-07T09:08:06.1781121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1682, in forward
2025-09-07T09:08:06.1781587Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T09:08:06.1782035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1670, in forward
2025-09-07T09:08:06.1782458Z     hidden_states = self.transform(hidden_states)
2025-09-07T09:08:06.1782987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1645, in forward
2025-09-07T09:08:06.1783403Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:08:06.1783541Z 
2025-09-07T09:08:06.1783648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:06.1784152Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2318, in torch_dynamo_resume_in_forward_at_2303
2025-09-07T09:08:06.1784646Z     prediction_scores = self.cls(sequence_output)
2025-09-07T09:08:06.1785058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1682, in forward
2025-09-07T09:08:06.1785497Z     prediction_scores = self.predictions(sequence_output)
2025-09-07T09:08:06.1785915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1671, in forward
2025-09-07T09:08:06.1786332Z     hidden_states = self.decoder(hidden_states)
2025-09-07T09:08:06.1786476Z 
2025-09-07T09:08:08.8335327Z 
2025-09-07T09:08:09.8013254Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:08:10.6888463Z running benchmark:   3% 1/30 [00:00<00:28,  1.03it/s]
2025-09-07T09:08:11.6287655Z running benchmark:   7% 2/30 [00:01<00:25,  1.09it/s]
2025-09-07T09:08:12.5228227Z running benchmark:  10% 3/30 [00:02<00:25,  1.08it/s]
2025-09-07T09:08:13.4858967Z running benchmark:  13% 4/30 [00:03<00:23,  1.09it/s]
2025-09-07T09:08:14.3775755Z running benchmark:  17% 5/30 [00:04<00:23,  1.07it/s]
2025-09-07T09:08:15.2658171Z running benchmark:  20% 6/30 [00:05<00:22,  1.09it/s]
2025-09-07T09:08:16.1477843Z running benchmark:  23% 7/30 [00:06<00:20,  1.10it/s]
2025-09-07T09:08:17.0319031Z running benchmark:  27% 8/30 [00:07<00:19,  1.11it/s]
2025-09-07T09:08:17.9856233Z running benchmark:  30% 9/30 [00:08<00:18,  1.12it/s]
2025-09-07T09:08:18.8729654Z running benchmark:  33% 10/30 [00:09<00:18,  1.09it/s]
2025-09-07T09:08:19.7591223Z running benchmark:  37% 11/30 [00:10<00:17,  1.10it/s]
2025-09-07T09:08:20.6404357Z running benchmark:  40% 12/30 [00:10<00:16,  1.11it/s]
2025-09-07T09:08:21.5941666Z running benchmark:  43% 13/30 [00:11<00:15,  1.12it/s]
2025-09-07T09:08:22.4789096Z running benchmark:  47% 14/30 [00:12<00:14,  1.10it/s]
2025-09-07T09:08:23.3620934Z running benchmark:  50% 15/30 [00:13<00:13,  1.11it/s]
2025-09-07T09:08:24.2497887Z running benchmark:  53% 16/30 [00:14<00:12,  1.11it/s]
2025-09-07T09:08:25.1364276Z running benchmark:  57% 17/30 [00:15<00:11,  1.12it/s]
2025-09-07T09:08:26.1101813Z running benchmark:  60% 18/30 [00:16<00:10,  1.12it/s]
2025-09-07T09:08:26.9878746Z running benchmark:  63% 19/30 [00:17<00:10,  1.09it/s]
2025-09-07T09:08:27.8728422Z running benchmark:  67% 20/30 [00:18<00:09,  1.10it/s]
2025-09-07T09:08:28.7553119Z running benchmark:  70% 21/30 [00:19<00:08,  1.11it/s]
2025-09-07T09:08:29.7129304Z running benchmark:  73% 22/30 [00:19<00:07,  1.12it/s]
2025-09-07T09:08:30.5990724Z running benchmark:  77% 23/30 [00:20<00:06,  1.10it/s]
2025-09-07T09:08:31.4851558Z running benchmark:  80% 24/30 [00:21<00:05,  1.10it/s]
2025-09-07T09:08:32.3621917Z running benchmark:  83% 25/30 [00:22<00:04,  1.11it/s]
2025-09-07T09:08:33.2482010Z running benchmark:  87% 26/30 [00:23<00:03,  1.12it/s]
2025-09-07T09:08:34.2026311Z running benchmark:  90% 27/30 [00:24<00:02,  1.12it/s]
2025-09-07T09:08:35.0879296Z running benchmark:  93% 28/30 [00:25<00:01,  1.10it/s]
2025-09-07T09:08:35.9730303Z running benchmark:  97% 29/30 [00:26<00:00,  1.11it/s]
2025-09-07T09:08:35.9732537Z running benchmark: 100% 30/30 [00:27<00:00,  1.11it/s]
2025-09-07T09:08:35.9734125Z running benchmark: 100% 30/30 [00:27<00:00,  1.11it/s]
2025-09-07T09:08:36.0777527Z 1.065x
2025-09-07T09:08:36.0781814Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:08:38.5963362Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:08:38.5964242Z   import pynvml  # type: ignore[import]
2025-09-07T09:08:40.6200885Z 
2025-09-07T09:08:43.0991796Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:08:43.0992267Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:08:43.1088445Z cpu  eval  hf_DistilBert                      
2025-09-07T09:08:47.0589032Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0589394Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0589612Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0589811Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0590071Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0590280Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0590519Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0590910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0591260Z     return mod(**inputs)
2025-09-07T09:08:47.0591690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0592143Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0592588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0593028Z     return self.transformer(
2025-09-07T09:08:47.0593472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0593903Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0594256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0594621Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0595048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0595465Z     sa_output = self.attention(
2025-09-07T09:08:47.0595865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T09:08:47.0596337Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T09:08:47.0596820Z 
2025-09-07T09:08:47.0596931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0597382Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0597710Z     return mod(**inputs)
2025-09-07T09:08:47.0598105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0598561Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0598963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0599373Z     return self.transformer(
2025-09-07T09:08:47.0599767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0600177Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0600596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0600961Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0601375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0601782Z     sa_output = self.attention(
2025-09-07T09:08:47.0602243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T09:08:47.0602702Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0602883Z 
2025-09-07T09:08:47.0603170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0603531Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0603848Z     return mod(**inputs)
2025-09-07T09:08:47.0604237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0613450Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0613938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0614377Z     return self.transformer(
2025-09-07T09:08:47.0614804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0615241Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0615611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0616010Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0616443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0616864Z     sa_output = self.attention(
2025-09-07T09:08:47.0617272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T09:08:47.0617817Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0618001Z 
2025-09-07T09:08:47.0618088Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0618322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0618686Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0619018Z     return mod(**inputs)
2025-09-07T09:08:47.0619414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0619830Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0620241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0620758Z     return self.transformer(
2025-09-07T09:08:47.0621161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0621632Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0621973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0622338Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0622755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0623172Z     sa_output = self.attention(
2025-09-07T09:08:47.0623559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T09:08:47.0624033Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:08:47.0624226Z 
2025-09-07T09:08:47.0624372Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0624743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0625069Z     return mod(**inputs)
2025-09-07T09:08:47.0625446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0625909Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0626312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0626721Z     return self.transformer(
2025-09-07T09:08:47.0627109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0627510Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0627860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0628240Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0628670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0629086Z     sa_output = self.attention(
2025-09-07T09:08:47.0629475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T09:08:47.0629891Z     attn_output = self.out_lin(attn_output)
2025-09-07T09:08:47.0630030Z 
2025-09-07T09:08:47.0630130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0630489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0630805Z     return mod(**inputs)
2025-09-07T09:08:47.0631189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0631606Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0632006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0632413Z     return self.transformer(
2025-09-07T09:08:47.0632795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0633203Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0633538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0633898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0634306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0634745Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0637146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0637743Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0638277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0638684Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0639091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T09:08:47.0639503Z     x = self.lin1(input)
2025-09-07T09:08:47.0639608Z 
2025-09-07T09:08:47.0639709Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0640068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0640379Z     return mod(**inputs)
2025-09-07T09:08:47.0640802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0641222Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0641621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0642030Z     return self.transformer(
2025-09-07T09:08:47.0642443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0642854Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0643192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0643551Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0643961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0644401Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0644838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0645371Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0645890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0646280Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0646684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T09:08:47.0647095Z     x = self.activation(x)
2025-09-07T09:08:47.0647417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:08:47.0647755Z     return self.act(input)
2025-09-07T09:08:47.0647854Z 
2025-09-07T09:08:47.0647952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0648296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0648612Z     return mod(**inputs)
2025-09-07T09:08:47.0648991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0649404Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0649795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0650199Z     return self.transformer(
2025-09-07T09:08:47.0650589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0651000Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0651397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0651791Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0652201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0652644Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0653080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0653609Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0654123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0654515Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0654966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T09:08:47.0655381Z     x = self.lin2(x)
2025-09-07T09:08:47.0655473Z 
2025-09-07T09:08:47.0655572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0655922Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0656269Z     return mod(**inputs)
2025-09-07T09:08:47.0656654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0657062Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0657451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0657927Z     return self.transformer(
2025-09-07T09:08:47.0658313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0658719Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0659046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0659399Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0659810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0660216Z     sa_output = self.attention(
2025-09-07T09:08:47.0660603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T09:08:47.0661054Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T09:08:47.0661237Z 
2025-09-07T09:08:47.0661334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0661683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0662003Z     return mod(**inputs)
2025-09-07T09:08:47.0662383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0662790Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0663198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0663613Z     return self.transformer(
2025-09-07T09:08:47.0664008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0664418Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0664752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0665113Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0665575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0666027Z     sa_output = self.attention(
2025-09-07T09:08:47.0666411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T09:08:47.0666866Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0667045Z 
2025-09-07T09:08:47.0667145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0667499Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0667815Z     return mod(**inputs)
2025-09-07T09:08:47.0668190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0668597Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0669046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0669461Z     return self.transformer(
2025-09-07T09:08:47.0669846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0670247Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0670618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0670977Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0671387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0671794Z     sa_output = self.attention(
2025-09-07T09:08:47.0672199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T09:08:47.0672667Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0672848Z 
2025-09-07T09:08:47.0672932Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0673167Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0673515Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0673844Z     return mod(**inputs)
2025-09-07T09:08:47.0674235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0674655Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0675067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0675477Z     return self.transformer(
2025-09-07T09:08:47.0675871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0676286Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0676634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0676988Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0677404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0677810Z     sa_output = self.attention(
2025-09-07T09:08:47.0678207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T09:08:47.0678676Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:08:47.0678859Z 
2025-09-07T09:08:47.0678960Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0679319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0679710Z     return mod(**inputs)
2025-09-07T09:08:47.0680095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0680509Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0680912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0681319Z     return self.transformer(
2025-09-07T09:08:47.0681703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0682108Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0682446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0682803Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0683246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0683652Z     sa_output = self.attention(
2025-09-07T09:08:47.0684037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T09:08:47.0684456Z     attn_output = self.out_lin(attn_output)
2025-09-07T09:08:47.0684621Z 
2025-09-07T09:08:47.0684723Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0685072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0685380Z     return mod(**inputs)
2025-09-07T09:08:47.0685755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0686159Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0686559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0686959Z     return self.transformer(
2025-09-07T09:08:47.0687346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0687752Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0688087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0688438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0688837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0689288Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0689730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0690275Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0690800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0691194Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0691606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T09:08:47.0692015Z     x = self.lin1(input)
2025-09-07T09:08:47.0692111Z 
2025-09-07T09:08:47.0692216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0692564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0692872Z     return mod(**inputs)
2025-09-07T09:08:47.0693247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0693756Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0694201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0694603Z     return self.transformer(
2025-09-07T09:08:47.0694997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0695405Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0695742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0696096Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0696496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0696930Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0697407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0697982Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0698500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0698921Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0699330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T09:08:47.0699738Z     x = self.activation(x)
2025-09-07T09:08:47.0700057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:08:47.0700393Z     return self.act(input)
2025-09-07T09:08:47.0700493Z 
2025-09-07T09:08:47.0700590Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0700938Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0701254Z     return mod(**inputs)
2025-09-07T09:08:47.0701634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0702039Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0702437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0702843Z     return self.transformer(
2025-09-07T09:08:47.0703446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0703863Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0704197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0704564Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0704968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0705415Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0705869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0706398Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0706913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0707305Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0707717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T09:08:47.0708201Z     x = self.lin2(x)
2025-09-07T09:08:47.0708294Z 
2025-09-07T09:08:47.0708440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0708791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0709107Z     return mod(**inputs)
2025-09-07T09:08:47.0709489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0709895Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0710295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0710701Z     return self.transformer(
2025-09-07T09:08:47.0711084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0711491Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0711870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0712232Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0712644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0713052Z     sa_output = self.attention(
2025-09-07T09:08:47.0713484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T09:08:47.0713953Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T09:08:47.0714140Z 
2025-09-07T09:08:47.0714237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0714586Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0714904Z     return mod(**inputs)
2025-09-07T09:08:47.0715284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0715699Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0716104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0716510Z     return self.transformer(
2025-09-07T09:08:47.0716901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0717306Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0717643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0717998Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0718407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0718832Z     sa_output = self.attention(
2025-09-07T09:08:47.0719217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T09:08:47.0719670Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0719851Z 
2025-09-07T09:08:47.0719946Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0720297Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0720608Z     return mod(**inputs)
2025-09-07T09:08:47.0720989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0721397Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0721794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0722248Z     return self.transformer(
2025-09-07T09:08:47.0722752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0723161Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0723497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0723854Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0724261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0724661Z     sa_output = self.attention(
2025-09-07T09:08:47.0725047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T09:08:47.0725499Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0725675Z 
2025-09-07T09:08:47.0725794Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0726016Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0726363Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0726675Z     return mod(**inputs)
2025-09-07T09:08:47.0727088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0727507Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0727898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0728308Z     return self.transformer(
2025-09-07T09:08:47.0728695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0729099Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0729435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0729785Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0730192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0730595Z     sa_output = self.attention(
2025-09-07T09:08:47.0730980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T09:08:47.0731434Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:08:47.0731622Z 
2025-09-07T09:08:47.0731715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0732056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0732369Z     return mod(**inputs)
2025-09-07T09:08:47.0732743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0733145Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0733539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0733940Z     return self.transformer(
2025-09-07T09:08:47.0734326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0734730Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0735054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0735401Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0735810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0736269Z     sa_output = self.attention(
2025-09-07T09:08:47.0736679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T09:08:47.0737098Z     attn_output = self.out_lin(attn_output)
2025-09-07T09:08:47.0737235Z 
2025-09-07T09:08:47.0737327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0737727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0738043Z     return mod(**inputs)
2025-09-07T09:08:47.0738415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0738838Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0739249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0739660Z     return self.transformer(
2025-09-07T09:08:47.0740095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0740513Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0740854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0741251Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0741663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0742107Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0742554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0743093Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0743621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0744021Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0744427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T09:08:47.0744834Z     x = self.lin1(input)
2025-09-07T09:08:47.0744944Z 
2025-09-07T09:08:47.0745041Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0745399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0745717Z     return mod(**inputs)
2025-09-07T09:08:47.0746092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0746498Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0746890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0747291Z     return self.transformer(
2025-09-07T09:08:47.0747690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0748108Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0748436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0748782Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0749178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0749604Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0750031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0750608Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0751169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0751581Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0751996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T09:08:47.0752440Z     x = self.activation(x)
2025-09-07T09:08:47.0752782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:08:47.0753118Z     return self.act(input)
2025-09-07T09:08:47.0753217Z 
2025-09-07T09:08:47.0753317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0753658Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0754013Z     return mod(**inputs)
2025-09-07T09:08:47.0754395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0754810Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0755203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0755635Z     return self.transformer(
2025-09-07T09:08:47.0756023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0756433Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0756773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0757123Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0757528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0757966Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0758403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0758936Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0759446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0759837Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0760246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T09:08:47.0760653Z     x = self.lin2(x)
2025-09-07T09:08:47.0760745Z 
2025-09-07T09:08:47.0760847Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0761185Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0761501Z     return mod(**inputs)
2025-09-07T09:08:47.0761875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0762282Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0762674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0763079Z     return self.transformer(
2025-09-07T09:08:47.0763461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0763867Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0764198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0764593Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0765037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0765445Z     sa_output = self.attention(
2025-09-07T09:08:47.0765841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T09:08:47.0766302Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T09:08:47.0766478Z 
2025-09-07T09:08:47.0766576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0766919Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0767230Z     return mod(**inputs)
2025-09-07T09:08:47.0767606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0768050Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0768447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0768848Z     return self.transformer(
2025-09-07T09:08:47.0769233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0769670Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0769995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0770355Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0770762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0771166Z     sa_output = self.attention(
2025-09-07T09:08:47.0771554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T09:08:47.0772003Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0772179Z 
2025-09-07T09:08:47.0772274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0772617Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0772930Z     return mod(**inputs)
2025-09-07T09:08:47.0773305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0773705Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0774100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0774505Z     return self.transformer(
2025-09-07T09:08:47.0774889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0775289Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0775620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0775972Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0776379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0776785Z     sa_output = self.attention(
2025-09-07T09:08:47.0777163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T09:08:47.0777663Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0777845Z 
2025-09-07T09:08:47.0777921Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0778143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0778547Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0778895Z     return mod(**inputs)
2025-09-07T09:08:47.0779269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0779676Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0780073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0780473Z     return self.transformer(
2025-09-07T09:08:47.0780854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0781258Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0781589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0781973Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0782375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0782784Z     sa_output = self.attention(
2025-09-07T09:08:47.0783203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T09:08:47.0783668Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:08:47.0783850Z 
2025-09-07T09:08:47.0783952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0784292Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0784603Z     return mod(**inputs)
2025-09-07T09:08:47.0784983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0785391Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0785779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0786186Z     return self.transformer(
2025-09-07T09:08:47.0786571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0786977Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0787306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0787655Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0788060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0788464Z     sa_output = self.attention(
2025-09-07T09:08:47.0788854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T09:08:47.0789274Z     attn_output = self.out_lin(attn_output)
2025-09-07T09:08:47.0789406Z 
2025-09-07T09:08:47.0789498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0789839Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0790152Z     return mod(**inputs)
2025-09-07T09:08:47.0790527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0790929Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0791327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0791730Z     return self.transformer(
2025-09-07T09:08:47.0792117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0792606Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0792933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0793283Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0793686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0794127Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0794567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0795099Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0795618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0796050Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0796462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T09:08:47.0796866Z     x = self.lin1(input)
2025-09-07T09:08:47.0796959Z 
2025-09-07T09:08:47.0797052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0797436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0797753Z     return mod(**inputs)
2025-09-07T09:08:47.0798129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0798532Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0798926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0799329Z     return self.transformer(
2025-09-07T09:08:47.0799725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0800132Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0800456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0800809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0801215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0801654Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0802103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0802637Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0803309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0803706Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0804112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T09:08:47.0804523Z     x = self.activation(x)
2025-09-07T09:08:47.0804833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:08:47.0805170Z     return self.act(input)
2025-09-07T09:08:47.0805274Z 
2025-09-07T09:08:47.0805368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0805714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0806025Z     return mod(**inputs)
2025-09-07T09:08:47.0806407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0806938Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0807341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0807752Z     return self.transformer(
2025-09-07T09:08:47.0808134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0808541Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0808877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0809236Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0809656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0810142Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0810583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0811125Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0811871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0812279Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0812686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T09:08:47.0813087Z     x = self.lin2(x)
2025-09-07T09:08:47.0813185Z 
2025-09-07T09:08:47.0813284Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0813637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0813951Z     return mod(**inputs)
2025-09-07T09:08:47.0814336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0814749Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0815153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0815562Z     return self.transformer(
2025-09-07T09:08:47.0815945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0816349Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0816693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0817055Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0817457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0817914Z     sa_output = self.attention(
2025-09-07T09:08:47.0818306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T09:08:47.0818765Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T09:08:47.0818945Z 
2025-09-07T09:08:47.0819051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0819392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0819710Z     return mod(**inputs)
2025-09-07T09:08:47.0820091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0820506Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0820902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0821386Z     return self.transformer(
2025-09-07T09:08:47.0821773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0822178Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0822514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0822868Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0823264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0823675Z     sa_output = self.attention(
2025-09-07T09:08:47.0824064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T09:08:47.0824553Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0824726Z 
2025-09-07T09:08:47.0824829Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0825165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0825479Z     return mod(**inputs)
2025-09-07T09:08:47.0825908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0826328Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0826721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0827126Z     return self.transformer(
2025-09-07T09:08:47.0827514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0827924Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0828259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0828607Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0829012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0829414Z     sa_output = self.attention(
2025-09-07T09:08:47.0829800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T09:08:47.0830246Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0830425Z 
2025-09-07T09:08:47.0830502Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0830725Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0831064Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0831384Z     return mod(**inputs)
2025-09-07T09:08:47.0831750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0832155Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0832549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0832955Z     return self.transformer(
2025-09-07T09:08:47.0833351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0833758Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0834095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0834448Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0834856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0835334Z     sa_output = self.attention(
2025-09-07T09:08:47.0835731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T09:08:47.0836189Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:08:47.0836369Z 
2025-09-07T09:08:47.0836473Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0836820Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0837133Z     return mod(**inputs)
2025-09-07T09:08:47.0837512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0837924Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0838357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0838769Z     return self.transformer(
2025-09-07T09:08:47.0839150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0839555Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0839921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0840286Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0840685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0841096Z     sa_output = self.attention(
2025-09-07T09:08:47.0841483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T09:08:47.0841906Z     attn_output = self.out_lin(attn_output)
2025-09-07T09:08:47.0842038Z 
2025-09-07T09:08:47.0842141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0842475Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0842787Z     return mod(**inputs)
2025-09-07T09:08:47.0843175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0843587Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0843988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0844386Z     return self.transformer(
2025-09-07T09:08:47.0844769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0845178Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0845511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0845857Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0846265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0846709Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0847148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0847685Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0848199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0848593Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0849050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T09:08:47.0849499Z     x = self.lin1(input)
2025-09-07T09:08:47.0849595Z 
2025-09-07T09:08:47.0849698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0850040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0850352Z     return mod(**inputs)
2025-09-07T09:08:47.0850727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0851141Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0851537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0851938Z     return self.transformer(
2025-09-07T09:08:47.0852359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0852776Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0853123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0853481Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0853928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0854383Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0854824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0855366Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0855881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0856280Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0856693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T09:08:47.0857107Z     x = self.activation(x)
2025-09-07T09:08:47.0857432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:08:47.0857832Z     return self.act(input)
2025-09-07T09:08:47.0857939Z 
2025-09-07T09:08:47.0858037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0858392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0858721Z     return mod(**inputs)
2025-09-07T09:08:47.0859114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0859528Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0859933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0860347Z     return self.transformer(
2025-09-07T09:08:47.0860734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0861139Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0861485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0861838Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0862246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0862688Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0863122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0863741Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0864263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0864658Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0865072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T09:08:47.0865470Z     x = self.lin2(x)
2025-09-07T09:08:47.0865573Z 
2025-09-07T09:08:47.0865673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0866021Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0866345Z     return mod(**inputs)
2025-09-07T09:08:47.0866767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0867182Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0867582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0867988Z     return self.transformer(
2025-09-07T09:08:47.0868408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0868817Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0869156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0869514Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0869924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0870336Z     sa_output = self.attention(
2025-09-07T09:08:47.0870724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
2025-09-07T09:08:47.0871179Z     q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
2025-09-07T09:08:47.0871365Z 
2025-09-07T09:08:47.0871462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0871813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0872125Z     return mod(**inputs)
2025-09-07T09:08:47.0872499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0872905Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0873305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0873716Z     return self.transformer(
2025-09-07T09:08:47.0874102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0874513Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0874848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0875229Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0875639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0876040Z     sa_output = self.attention(
2025-09-07T09:08:47.0876431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward
2025-09-07T09:08:47.0876886Z     k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0877056Z 
2025-09-07T09:08:47.0877234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0877581Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0877932Z     return mod(**inputs)
2025-09-07T09:08:47.0878308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0878724Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0879128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0879537Z     return self.transformer(
2025-09-07T09:08:47.0879920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0880326Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0880659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0881072Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0881482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0881885Z     sa_output = self.attention(
2025-09-07T09:08:47.0882307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward
2025-09-07T09:08:47.0882765Z     v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
2025-09-07T09:08:47.0882945Z 
2025-09-07T09:08:47.0883021Z cudagraph partition due to non gpu ops
2025-09-07T09:08:47.0883249Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0883588Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0883903Z     return mod(**inputs)
2025-09-07T09:08:47.0884279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0884693Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0885089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0885493Z     return self.transformer(
2025-09-07T09:08:47.0885888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0886289Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0886619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0886968Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0887374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0887781Z     sa_output = self.attention(
2025-09-07T09:08:47.0888168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward
2025-09-07T09:08:47.0888632Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:08:47.0888816Z 
2025-09-07T09:08:47.0888911Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0889253Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0889571Z     return mod(**inputs)
2025-09-07T09:08:47.0889946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0890354Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0890744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0891150Z     return self.transformer(
2025-09-07T09:08:47.0891577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0892022Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0892349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0892701Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0893109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
2025-09-07T09:08:47.0893510Z     sa_output = self.attention(
2025-09-07T09:08:47.0893898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward
2025-09-07T09:08:47.0894313Z     attn_output = self.out_lin(attn_output)
2025-09-07T09:08:47.0894459Z 
2025-09-07T09:08:47.0894552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0894925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0895241Z     return mod(**inputs)
2025-09-07T09:08:47.0895613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0896013Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0896440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0896850Z     return self.transformer(
2025-09-07T09:08:47.0897234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0897689Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0898016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0898369Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0898772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0899214Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0905709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0906382Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0906940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0907347Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0907775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk
2025-09-07T09:08:47.0908198Z     x = self.lin1(input)
2025-09-07T09:08:47.0908311Z 
2025-09-07T09:08:47.0908421Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0908783Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0909124Z     return mod(**inputs)
2025-09-07T09:08:47.0909533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0909956Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0910371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0910777Z     return self.transformer(
2025-09-07T09:08:47.0911167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0911578Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0912033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0912452Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0912865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0913307Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0913748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0914291Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0914811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0915206Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0915664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk
2025-09-07T09:08:47.0916078Z     x = self.activation(x)
2025-09-07T09:08:47.0916396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:08:47.0916721Z     return self.act(input)
2025-09-07T09:08:47.0916821Z 
2025-09-07T09:08:47.0916971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0917329Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0917647Z     return mod(**inputs)
2025-09-07T09:08:47.0918023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward
2025-09-07T09:08:47.0918429Z     dlbrt_output = self.distilbert(
2025-09-07T09:08:47.0918827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
2025-09-07T09:08:47.0919226Z     return self.transformer(
2025-09-07T09:08:47.0919610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
2025-09-07T09:08:47.0920015Z     layer_outputs = layer_module(
2025-09-07T09:08:47.0920363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:08:47.0920720Z     return super().__call__(*args, **kwargs)
2025-09-07T09:08:47.0921125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward
2025-09-07T09:08:47.0921564Z     ffn_output = self.ffn(sa_output)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0921994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward
2025-09-07T09:08:47.0922528Z     return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input)
2025-09-07T09:08:47.0923050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:08:47.0923443Z     return forward_fn(*input_tensors)
2025-09-07T09:08:47.0923849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk
2025-09-07T09:08:47.0924244Z     x = self.lin2(x)
2025-09-07T09:08:47.0924342Z 
2025-09-07T09:08:47.0924437Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0924782Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0925091Z     return mod(**inputs)
2025-09-07T09:08:47.0925464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 836, in forward
2025-09-07T09:08:47.0925967Z     prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
2025-09-07T09:08:47.0926267Z 
2025-09-07T09:08:47.0926360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:08:47.0926699Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:08:47.0927029Z     return mod(**inputs)
2025-09-07T09:08:47.0927393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 839, in forward
2025-09-07T09:08:47.0927902Z     prediction_logits = self.vocab_projector(prediction_logits)  # (bs, seq_length, vocab_size)
2025-09-07T09:08:47.0928141Z 
2025-09-07T09:08:53.1812771Z 
2025-09-07T09:08:53.2963404Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:08:53.4087285Z running benchmark:   7% 2/30 [00:00<00:01, 17.41it/s]
2025-09-07T09:08:53.5188848Z running benchmark:  13% 4/30 [00:00<00:01, 17.64it/s]
2025-09-07T09:08:53.6288817Z running benchmark:  20% 6/30 [00:00<00:01, 17.87it/s]
2025-09-07T09:08:53.7385573Z running benchmark:  27% 8/30 [00:00<00:01, 17.99it/s]
2025-09-07T09:08:53.9196765Z running benchmark:  33% 10/30 [00:00<00:01, 18.08it/s]
2025-09-07T09:08:54.0294407Z running benchmark:  40% 12/30 [00:00<00:01, 14.86it/s]
2025-09-07T09:08:54.1395832Z running benchmark:  47% 14/30 [00:00<00:01, 15.81it/s]
2025-09-07T09:08:54.2522090Z running benchmark:  53% 16/30 [00:00<00:00, 16.49it/s]
2025-09-07T09:08:54.3626135Z running benchmark:  60% 18/30 [00:01<00:00, 16.87it/s]
2025-09-07T09:08:54.4725946Z running benchmark:  67% 20/30 [00:01<00:00, 17.23it/s]
2025-09-07T09:08:54.5836298Z running benchmark:  73% 22/30 [00:01<00:00, 17.52it/s]
2025-09-07T09:08:54.6953017Z running benchmark:  80% 24/30 [00:01<00:00, 17.66it/s]
2025-09-07T09:08:54.8065595Z running benchmark:  87% 26/30 [00:01<00:00, 17.74it/s]
2025-09-07T09:08:54.9158937Z running benchmark:  93% 28/30 [00:01<00:00, 17.81it/s]
2025-09-07T09:08:54.9159313Z running benchmark: 100% 30/30 [00:01<00:00, 17.95it/s]
2025-09-07T09:08:54.9159663Z running benchmark: 100% 30/30 [00:01<00:00, 17.29it/s]
2025-09-07T09:08:54.9905389Z 1.150x
2025-09-07T09:08:54.9909452Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:08:57.3086539Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:08:57.3087524Z   import pynvml  # type: ignore[import]
2025-09-07T09:08:59.3271319Z 
2025-09-07T09:09:03.8340277Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:09:03.8340623Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:09:03.8498323Z cpu  eval  hf_GPT2                            
2025-09-07T09:09:09.3770793Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3771084Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3771335Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3771548Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3771771Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3771960Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3772160Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3772360Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3772558Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3772759Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3772962Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3773153Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3773413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3773798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3774145Z     return mod(**inputs)
2025-09-07T09:09:09.3774558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3775300Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3775799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3776200Z     outputs = block(
2025-09-07T09:09:09.3776536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3776906Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3777291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3777732Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3778106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3778509Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3779011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3779442Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3779817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.3780322Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.3780875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3781291Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3781466Z 
2025-09-07T09:09:09.3781555Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3781759Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3781998Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3782189Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3782420Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3782780Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3783117Z     return mod(**inputs)
2025-09-07T09:09:09.3783486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3783891Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3784281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3784668Z     outputs = block(
2025-09-07T09:09:09.3784987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3785339Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3785718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3786096Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3786462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3786862Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3787233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3787601Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3787964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3788366Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3788805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.3789300Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.3789540Z 
2025-09-07T09:09:09.3789642Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3790033Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3790353Z     return mod(**inputs)
2025-09-07T09:09:09.3790703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3791104Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3791487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3791858Z     outputs = block(
2025-09-07T09:09:09.3792170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3792524Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3792926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3793295Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3793655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3794047Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3794472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3794845Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3795201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3795596Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3796026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.3796490Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.3796660Z 
2025-09-07T09:09:09.3796757Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3797111Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3797430Z     return mod(**inputs)
2025-09-07T09:09:09.3797774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3798169Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3798544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3798915Z     outputs = block(
2025-09-07T09:09:09.3799216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3799575Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3799945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3800322Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3800678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3801062Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3801447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3801816Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3802169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.3802549Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.3802896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3803455Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3803738Z 
2025-09-07T09:09:09.3803890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3804239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3804546Z     return mod(**inputs)
2025-09-07T09:09:09.3804901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3805295Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3805673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3806039Z     outputs = block(
2025-09-07T09:09:09.3806348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3806706Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3807121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3807497Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3807855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3808252Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3808705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.3809104Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.3809459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3809852Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3810029Z 
2025-09-07T09:09:09.3810131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3810494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3810817Z     return mod(**inputs)
2025-09-07T09:09:09.3811177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3811567Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3811961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3812335Z     outputs = block(
2025-09-07T09:09:09.3812652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3813006Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3813375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3813746Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3814121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3814528Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3814922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.3815303Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.3815649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.3816107Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.3816344Z 
2025-09-07T09:09:09.3816449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3816799Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3817118Z     return mod(**inputs)
2025-09-07T09:09:09.3817570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3818012Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3818401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3818771Z     outputs = block(
2025-09-07T09:09:09.3819092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3819455Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3819832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3820199Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3820568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3821017Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3821419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.3821815Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.3822166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3822602Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3822785Z 
2025-09-07T09:09:09.3822884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3823246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3823567Z     return mod(**inputs)
2025-09-07T09:09:09.3823920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3824314Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3824706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3825078Z     outputs = block(
2025-09-07T09:09:09.3825389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3825745Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3826118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3826487Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3826848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3827234Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3827612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3827983Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3828344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.3828838Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.3829296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3829691Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3829864Z 
2025-09-07T09:09:09.3829941Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3830146Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3830333Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3830526Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3830749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3831102Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3831498Z     return mod(**inputs)
2025-09-07T09:09:09.3831844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3832238Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3832620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3832988Z     outputs = block(
2025-09-07T09:09:09.3833294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3833650Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3834020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3834392Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3834786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3835173Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3835559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3835935Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3836334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3836736Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3837186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.3837666Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.3837853Z 
2025-09-07T09:09:09.3837951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3838309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3838622Z     return mod(**inputs)
2025-09-07T09:09:09.3838978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3839373Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3839757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3840121Z     outputs = block(
2025-09-07T09:09:09.3840426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3840783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3841163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3841534Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3841889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3842278Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3842654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3843017Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3843373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3843757Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3844197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.3844651Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.3844810Z 
2025-09-07T09:09:09.3844914Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3845307Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3845653Z     return mod(**inputs)
2025-09-07T09:09:09.3846006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3846400Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3846782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3847142Z     outputs = block(
2025-09-07T09:09:09.3847451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3847809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3848180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3848589Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3848938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3849328Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3849705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3850121Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3850476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.3850860Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.3851213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3851603Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3851768Z 
2025-09-07T09:09:09.3851869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3852218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3852542Z     return mod(**inputs)
2025-09-07T09:09:09.3852895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3853291Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3853680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3854041Z     outputs = block(
2025-09-07T09:09:09.3854350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3854707Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3855073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3855451Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3855802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3856205Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3856605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.3856997Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.3857345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3857778Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3857952Z 
2025-09-07T09:09:09.3858050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3858398Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3858764Z     return mod(**inputs)
2025-09-07T09:09:09.3859112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3859547Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3859931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3860304Z     outputs = block(
2025-09-07T09:09:09.3860610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3860966Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3861373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3861768Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3862129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3862564Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3862968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.3863347Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.3863691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.3864173Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.3864409Z 
2025-09-07T09:09:09.3864511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3864868Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3865185Z     return mod(**inputs)
2025-09-07T09:09:09.3865536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3865934Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3866303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3866670Z     outputs = block(
2025-09-07T09:09:09.3866980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3867341Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3867702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3868067Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3868421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3868820Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3869223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.3869610Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.3869961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3870357Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3870523Z 
2025-09-07T09:09:09.3870627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3870976Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3871289Z     return mod(**inputs)
2025-09-07T09:09:09.3871637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3872022Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3872398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3872837Z     outputs = block(
2025-09-07T09:09:09.3873146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3873500Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3873874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3874242Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3874595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3874986Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3875363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3875731Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3876111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.3876608Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.3877073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3877498Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3877662Z 
2025-09-07T09:09:09.3877749Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3877949Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3878145Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3878336Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3878562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3878904Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3879223Z     return mod(**inputs)
2025-09-07T09:09:09.3879576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3879964Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3880339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3880700Z     outputs = block(
2025-09-07T09:09:09.3881010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3881358Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3881724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3882090Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3882452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3882844Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3883228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3883603Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3883960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3884362Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3884824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.3885309Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.3885489Z 
2025-09-07T09:09:09.3885592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3885939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3886304Z     return mod(**inputs)
2025-09-07T09:09:09.3886693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3887088Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3887458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3887839Z     outputs = block(
2025-09-07T09:09:09.3888151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3888505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3888873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3889237Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3889625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3890022Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3890404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3890767Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3891154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3891564Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3892011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.3892471Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.3892631Z 
2025-09-07T09:09:09.3892738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3893089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3893411Z     return mod(**inputs)
2025-09-07T09:09:09.3893786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3894182Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3894565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3894932Z     outputs = block(
2025-09-07T09:09:09.3895246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3895606Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3895978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3896336Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3896693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3897079Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3897501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3897861Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3898221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.3898605Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.3898954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3899350Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3899519Z 
2025-09-07T09:09:09.3899618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3899977Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3900375Z     return mod(**inputs)
2025-09-07T09:09:09.3900731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3901120Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3901495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3901862Z     outputs = block(
2025-09-07T09:09:09.3902173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3902526Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3902883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3903416Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3903844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3904266Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3904669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.3905046Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.3905449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3905850Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3906021Z 
2025-09-07T09:09:09.3906128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3906481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3906795Z     return mod(**inputs)
2025-09-07T09:09:09.3907152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3907546Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3907925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3908284Z     outputs = block(
2025-09-07T09:09:09.3908599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3908954Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3909326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3909694Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3910051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3910458Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3910860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.3911244Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.3911577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.3912023Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.3912258Z 
2025-09-07T09:09:09.3912354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3912707Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3913027Z     return mod(**inputs)
2025-09-07T09:09:09.3913371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3913757Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3914196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3914619Z     outputs = block(
2025-09-07T09:09:09.3914942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3915292Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3915664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3916041Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3916401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3916796Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3917191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.3917623Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.3917983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3918381Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3918549Z 
2025-09-07T09:09:09.3918646Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3919025Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3919342Z     return mod(**inputs)
2025-09-07T09:09:09.3919697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3920084Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3920453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3920820Z     outputs = block(
2025-09-07T09:09:09.3921134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3921486Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3921847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3922216Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3922574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:09.3922983Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:09.3923139Z 
2025-09-07T09:09:09.3923236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3923572Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3923890Z     return mod(**inputs)
2025-09-07T09:09:09.3924237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3924627Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3925005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3925365Z     outputs = block(
2025-09-07T09:09:09.3925668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3926018Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3926398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3926761Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3927122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3927515Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3927941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3928346Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3928696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.3929187Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.3929644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3930033Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3930200Z 
2025-09-07T09:09:09.3930407Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3930607Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3930812Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3931002Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3931256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3931599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3931920Z     return mod(**inputs)
2025-09-07T09:09:09.3932269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3933390Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3933818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3934205Z     outputs = block(
2025-09-07T09:09:09.3934523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3934878Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3935256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3935621Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3935987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3936379Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3936762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3937132Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3937549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3937950Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3938398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.3938881Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.3939065Z 
2025-09-07T09:09:09.3939171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3939520Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3939839Z     return mod(**inputs)
2025-09-07T09:09:09.3940197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3940585Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3940959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3941330Z     outputs = block(
2025-09-07T09:09:09.3941638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3941993Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3942362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3942814Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3943178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3943564Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3943946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3944304Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3944667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3945065Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3945510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.3946007Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.3946169Z 
2025-09-07T09:09:09.3946264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3946612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3946922Z     return mod(**inputs)
2025-09-07T09:09:09.3947305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3947697Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3948064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3948437Z     outputs = block(
2025-09-07T09:09:09.3948740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3949094Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3949456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3949821Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3950168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3950552Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3950923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3951282Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3951634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.3952017Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.3952365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3952768Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3952941Z 
2025-09-07T09:09:09.3953037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3953399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3953711Z     return mod(**inputs)
2025-09-07T09:09:09.3954064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3954464Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3954837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3955209Z     outputs = block(
2025-09-07T09:09:09.3955521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3955874Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3956277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3956764Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3957118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3957519Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3957917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.3958290Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.3958640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3959027Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3959193Z 
2025-09-07T09:09:09.3959295Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3959692Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3960022Z     return mod(**inputs)
2025-09-07T09:09:09.3960370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3960757Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3961171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3961535Z     outputs = block(
2025-09-07T09:09:09.3961843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3962196Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3962560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3962920Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3963274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3963674Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3964073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.3964455Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.3964787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.3965243Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.3965482Z 
2025-09-07T09:09:09.3965575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3965923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3966242Z     return mod(**inputs)
2025-09-07T09:09:09.3966583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3966977Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3967348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3967717Z     outputs = block(
2025-09-07T09:09:09.3968015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3968362Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3968727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3969096Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3969449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.3969896Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.3970323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.3970713Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.3971069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3971457Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3971625Z 
2025-09-07T09:09:09.3971719Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3972062Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3972376Z     return mod(**inputs)
2025-09-07T09:09:09.3972722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3973142Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3973514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3973877Z     outputs = block(
2025-09-07T09:09:09.3974182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3974560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3974922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3975283Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3975635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3976022Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3976399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3976754Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3977101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.3977646Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.3978105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.3978509Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.3978674Z 
2025-09-07T09:09:09.3978750Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3978950Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3979144Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3979334Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.3979541Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3979889Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3980204Z     return mod(**inputs)
2025-09-07T09:09:09.3980554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3980945Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3981341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3981706Z     outputs = block(
2025-09-07T09:09:09.3982011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3982366Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3982731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3983103Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3983512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.3983953Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.3984339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3984706Z     return func(*args, **kwargs)
2025-09-07T09:09:09.3985064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.3985458Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.3985905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.3986377Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.3986561Z 
2025-09-07T09:09:09.3986694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.3996148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.3996563Z     return mod(**inputs)
2025-09-07T09:09:09.3996971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.3997400Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.3997893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.3998288Z     outputs = block(
2025-09-07T09:09:09.3998615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.3998986Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.3999375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.3999765Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4000141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4000552Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4000937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4001309Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4001670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4002077Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4002524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4002987Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4003330Z 
2025-09-07T09:09:09.4003439Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4003801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4004121Z     return mod(**inputs)
2025-09-07T09:09:09.4004489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4004897Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4005299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4005689Z     outputs = block(
2025-09-07T09:09:09.4006004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4006364Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4006743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4007210Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4007632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4008037Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4008430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4008803Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4009168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4009545Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4009901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4010304Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4010479Z 
2025-09-07T09:09:09.4010633Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4011004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4011319Z     return mod(**inputs)
2025-09-07T09:09:09.4011691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4012136Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4012527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4012888Z     outputs = block(
2025-09-07T09:09:09.4013207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4013566Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4013944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4014314Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4014667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4015070Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4015478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4015859Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4016207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4016593Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4016770Z 
2025-09-07T09:09:09.4016866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4017216Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4017589Z     return mod(**inputs)
2025-09-07T09:09:09.4017936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4018326Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4018703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4019071Z     outputs = block(
2025-09-07T09:09:09.4019378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4019722Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4020093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4020459Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4020819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4021299Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4021697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4022074Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4022418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4022870Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4023109Z 
2025-09-07T09:09:09.4023207Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4023563Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4023879Z     return mod(**inputs)
2025-09-07T09:09:09.4024271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4024675Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4025048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4025416Z     outputs = block(
2025-09-07T09:09:09.4025760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4026122Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4026494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4026859Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4027216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4027619Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4028023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4028414Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4028780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4029179Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4029344Z 
2025-09-07T09:09:09.4029447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4029794Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4030105Z     return mod(**inputs)
2025-09-07T09:09:09.4030450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4030840Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4031221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4031588Z     outputs = block(
2025-09-07T09:09:09.4031886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4032236Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4032609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4032978Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4033325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:09.4033731Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:09.4033894Z 
2025-09-07T09:09:09.4033987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4034333Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4034690Z     return mod(**inputs)
2025-09-07T09:09:09.4035067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4035458Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4035834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4036195Z     outputs = block(
2025-09-07T09:09:09.4036497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4036857Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4037226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4037593Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4037981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4038369Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4038744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4039105Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4039482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4039977Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4040435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4040825Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4040997Z 
2025-09-07T09:09:09.4041077Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4041284Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4041467Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4041656Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4041871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4042215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4042524Z     return mod(**inputs)
2025-09-07T09:09:09.4042869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4043265Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4043639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4044004Z     outputs = block(
2025-09-07T09:09:09.4044302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4044651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4045020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4045387Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4045738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4046118Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4046492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4046852Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4047201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4047590Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4048078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4048685Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4048889Z 
2025-09-07T09:09:09.4048992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4049376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4049684Z     return mod(**inputs)
2025-09-07T09:09:09.4050049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4050440Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4050821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4051182Z     outputs = block(
2025-09-07T09:09:09.4051494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4051883Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4052260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4052624Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4053012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4053431Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4053814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4054183Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4054566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4054957Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4055411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4055884Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4056044Z 
2025-09-07T09:09:09.4056141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4056490Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4056803Z     return mod(**inputs)
2025-09-07T09:09:09.4057152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4057586Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4057963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4058323Z     outputs = block(
2025-09-07T09:09:09.4058634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4058980Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4059346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4059708Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4060063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4060441Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4060816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4061176Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4061523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4061902Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4062291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4062718Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4062893Z 
2025-09-07T09:09:09.4062987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4063336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4063645Z     return mod(**inputs)
2025-09-07T09:09:09.4063988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4064373Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4064744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4065099Z     outputs = block(
2025-09-07T09:09:09.4065429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4065777Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4066137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4066495Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4066872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4067281Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4067684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4068058Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4068396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4068773Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4068955Z 
2025-09-07T09:09:09.4069051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4069389Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4069709Z     return mod(**inputs)
2025-09-07T09:09:09.4070076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4070453Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4070818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4071175Z     outputs = block(
2025-09-07T09:09:09.4071474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4071817Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4072175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4072531Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4072872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4073259Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4073646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4074013Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4074339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4074786Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4075021Z 
2025-09-07T09:09:09.4075120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4075498Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4075841Z     return mod(**inputs)
2025-09-07T09:09:09.4076183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4076566Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4076933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4077283Z     outputs = block(
2025-09-07T09:09:09.4077579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4077915Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4078270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4078625Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4079005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4079400Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4079788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4080204Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4080550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4080932Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4081102Z 
2025-09-07T09:09:09.4081195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4081533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4081836Z     return mod(**inputs)
2025-09-07T09:09:09.4082173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4082556Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4082925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4083278Z     outputs = block(
2025-09-07T09:09:09.4083576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4083924Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4084281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4084638Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4084990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4085369Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4085737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4086089Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4086430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4086912Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4087359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4087734Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4087897Z 
2025-09-07T09:09:09.4087969Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4088163Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4088343Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4088572Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4088810Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4089148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4089451Z     return mod(**inputs)
2025-09-07T09:09:09.4089798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4090180Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4090553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4090906Z     outputs = block(
2025-09-07T09:09:09.4091201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4091541Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4091934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4092297Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4092643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4093014Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4093411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4093773Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4094119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4094501Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4094941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4095418Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4095594Z 
2025-09-07T09:09:09.4095689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4096030Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4096334Z     return mod(**inputs)
2025-09-07T09:09:09.4096686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4097067Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4097525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4097887Z     outputs = block(
2025-09-07T09:09:09.4098180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4098529Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4098893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4099253Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4099597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4099969Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4100339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4100730Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4101072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4101454Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4101889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4102404Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4102592Z 
2025-09-07T09:09:09.4102689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4103023Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4103492Z     return mod(**inputs)
2025-09-07T09:09:09.4103850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4104238Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4104615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4104971Z     outputs = block(
2025-09-07T09:09:09.4105280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4105694Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4106067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4106436Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4106790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4107242Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4107621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4107985Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4108342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4108719Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4109062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4109460Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4109631Z 
2025-09-07T09:09:09.4109735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4110081Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4110399Z     return mod(**inputs)
2025-09-07T09:09:09.4110752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4111141Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4111530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4111907Z     outputs = block(
2025-09-07T09:09:09.4112222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4112580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4112964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4113333Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4113686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4114084Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4114476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4114849Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4115182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4115568Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4115739Z 
2025-09-07T09:09:09.4115897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4116290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4116601Z     return mod(**inputs)
2025-09-07T09:09:09.4116940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4117325Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4117696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4118054Z     outputs = block(
2025-09-07T09:09:09.4118354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4118700Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4119064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4119666Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4120020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4120411Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4120835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4121224Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4121568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4122015Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4122254Z 
2025-09-07T09:09:09.4122352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4122703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4123021Z     return mod(**inputs)
2025-09-07T09:09:09.4123373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4123754Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4124128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4124488Z     outputs = block(
2025-09-07T09:09:09.4124793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4125143Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4125503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4125867Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4126221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4126617Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4127007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4127382Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4127730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4128119Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4128288Z 
2025-09-07T09:09:09.4128384Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4128722Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4129029Z     return mod(**inputs)
2025-09-07T09:09:09.4129367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4129798Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4130207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4130565Z     outputs = block(
2025-09-07T09:09:09.4130870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4131221Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4131584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4131946Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4132291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:09.4132694Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:09.4132854Z 
2025-09-07T09:09:09.4132992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4133338Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4133640Z     return mod(**inputs)
2025-09-07T09:09:09.4133982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4134402Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4134773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4135129Z     outputs = block(
2025-09-07T09:09:09.4135426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4135767Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4136123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4136481Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4136823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4137202Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4137618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4137979Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4138324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4138804Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4139253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4139633Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4139800Z 
2025-09-07T09:09:09.4139877Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4140069Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4140247Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4140425Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4140632Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4140970Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4141272Z     return mod(**inputs)
2025-09-07T09:09:09.4141611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4141993Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4142357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4142765Z     outputs = block(
2025-09-07T09:09:09.4143063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4143450Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4143810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4144184Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4144530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4144909Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4145278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4145631Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4145971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4146388Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4146828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4147302Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4147479Z 
2025-09-07T09:09:09.4147622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4147963Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4148276Z     return mod(**inputs)
2025-09-07T09:09:09.4148616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4148997Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4149365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4149726Z     outputs = block(
2025-09-07T09:09:09.4150027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4150377Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4150740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4151102Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4151458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4151843Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4152217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4152575Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4152818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4152906Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4153203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4153302Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4153306Z 
2025-09-07T09:09:09.4153411Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4153608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4153678Z     return mod(**inputs)
2025-09-07T09:09:09.4153929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4154012Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4154258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4154359Z     outputs = block(
2025-09-07T09:09:09.4154613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4154682Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4154920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4154986Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4155226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4155306Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4155537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4155600Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4155871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4155951Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4156171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4156281Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4156284Z 
2025-09-07T09:09:09.4156415Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4156605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4159696Z     return mod(**inputs)
2025-09-07T09:09:09.4160012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4160095Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4160364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4160426Z     outputs = block(
2025-09-07T09:09:09.4160653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4160733Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4160978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4161046Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4161291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4161421Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4161665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4161743Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4161960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4162083Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4162087Z 
2025-09-07T09:09:09.4162190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4162387Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4162452Z     return mod(**inputs)
2025-09-07T09:09:09.4162705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4162790Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4163034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4163091Z     outputs = block(
2025-09-07T09:09:09.4163314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4163440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4163721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4163785Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4164030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4164128Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4164370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4164446Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4164649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4164867Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4164874Z 
2025-09-07T09:09:09.4164975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4165166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4165227Z     return mod(**inputs)
2025-09-07T09:09:09.4165480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4165558Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4165947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4166004Z     outputs = block(
2025-09-07T09:09:09.4166225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4166296Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4166543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4166606Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4166852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4166944Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4167182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4167266Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4167483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4167600Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4167604Z 
2025-09-07T09:09:09.4167698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4167887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4167949Z     return mod(**inputs)
2025-09-07T09:09:09.4168195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4168270Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4168508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4168567Z     outputs = block(
2025-09-07T09:09:09.4168782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4168848Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4169085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4169145Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4169434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4169552Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4169797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4169865Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4170106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4170294Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4170507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4170615Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4170622Z 
2025-09-07T09:09:09.4170697Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4170801Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4170879Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4170946Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4171038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4171228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4171286Z     return mod(**inputs)
2025-09-07T09:09:09.4171537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4171661Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4171910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4171967Z     outputs = block(
2025-09-07T09:09:09.4172185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4172263Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4172502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4172571Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4172810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4172887Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4173126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4173186Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4173432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4173520Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4173820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4173947Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4173950Z 
2025-09-07T09:09:09.4174042Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4174236Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4174293Z     return mod(**inputs)
2025-09-07T09:09:09.4174549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4174622Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4174862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4174922Z     outputs = block(
2025-09-07T09:09:09.4175138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4175291Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4175527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4175589Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4175833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4175909Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4176151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4176210Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4176456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4176539Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4176859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4176971Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4176974Z 
2025-09-07T09:09:09.4177067Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4177262Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4177322Z     return mod(**inputs)
2025-09-07T09:09:09.4177674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4177758Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4178000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4178061Z     outputs = block(
2025-09-07T09:09:09.4178277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4178346Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4178587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4178646Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4178885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4178958Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4179192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4179249Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4179486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4179560Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4179769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4179873Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4179876Z 
2025-09-07T09:09:09.4179965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4180148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4180206Z     return mod(**inputs)
2025-09-07T09:09:09.4180449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4180520Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4180759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4180814Z     outputs = block(
2025-09-07T09:09:09.4181081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4181186Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4181429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4181488Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4181736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4181830Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4182072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4182145Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4182360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4182506Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4182511Z 
2025-09-07T09:09:09.4182606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4182794Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4182854Z     return mod(**inputs)
2025-09-07T09:09:09.4183101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4183176Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4183462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4183520Z     outputs = block(
2025-09-07T09:09:09.4183734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4183800Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4184044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4184103Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4184348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4184434Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4184679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4184752Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4184965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4185139Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4185143Z 
2025-09-07T09:09:09.4185241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4185431Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4185491Z     return mod(**inputs)
2025-09-07T09:09:09.4185743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4185813Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4186059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4186112Z     outputs = block(
2025-09-07T09:09:09.4186328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4186407Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4186643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4186759Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4187035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4187127Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4187372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4187449Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4187668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4187777Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4187781Z 
2025-09-07T09:09:09.4187874Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4188063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4188156Z     return mod(**inputs)
2025-09-07T09:09:09.4188430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4188508Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4188759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4188816Z     outputs = block(
2025-09-07T09:09:09.4189038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4189149Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4189391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4189459Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4189700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:09.4189800Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:09.4189809Z 
2025-09-07T09:09:09.4189899Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4190083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4190148Z     return mod(**inputs)
2025-09-07T09:09:09.4190393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4190477Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4190717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4190774Z     outputs = block(
2025-09-07T09:09:09.4190994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4191063Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4191308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4191371Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4191610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4191695Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4191926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4191989Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4192228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4192407Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4192667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4192807Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4192810Z 
2025-09-07T09:09:09.4192892Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4192960Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4193032Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4193098Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4193188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4193379Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4193436Z     return mod(**inputs)
2025-09-07T09:09:09.4193689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4193763Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4194039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4194102Z     outputs = block(
2025-09-07T09:09:09.4194316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4194390Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4194626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4194686Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4194973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4195052Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4195287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4195347Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4195598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4195689Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4195981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4196107Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4196111Z 
2025-09-07T09:09:09.4196200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4196407Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4196467Z     return mod(**inputs)
2025-09-07T09:09:09.4196712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4196791Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4197032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4197094Z     outputs = block(
2025-09-07T09:09:09.4197307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4197374Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4197612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4197679Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4197924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4197994Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4198235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4198353Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4198627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4198720Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4199010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4199116Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4199120Z 
2025-09-07T09:09:09.4199208Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4199396Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4199463Z     return mod(**inputs)
2025-09-07T09:09:09.4199710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4199826Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4200071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4200133Z     outputs = block(
2025-09-07T09:09:09.4200346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4200414Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4200656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4200717Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4201002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4201080Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4201320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4201388Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4201632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4201712Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4201925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4202027Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4202035Z 
2025-09-07T09:09:09.4202124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4202308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4202371Z     return mod(**inputs)
2025-09-07T09:09:09.4202615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4202697Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4202940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4202996Z     outputs = block(
2025-09-07T09:09:09.4203375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4203453Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4203702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4203762Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4204012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4204116Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4204361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4204561Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4204777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4204884Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4204894Z 
2025-09-07T09:09:09.4204987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4205179Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4205240Z     return mod(**inputs)
2025-09-07T09:09:09.4205490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4205573Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4205814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4205919Z     outputs = block(
2025-09-07T09:09:09.4206147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4206215Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4206462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4206521Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4206763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4206921Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4207167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4207244Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4207455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4207639Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4207642Z 
2025-09-07T09:09:09.4207735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4207922Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4207988Z     return mod(**inputs)
2025-09-07T09:09:09.4208237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4208318Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4208559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4208615Z     outputs = block(
2025-09-07T09:09:09.4208835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4208904Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4209148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4209207Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4209444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4209539Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4209778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4209859Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4210067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4210176Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4210218Z 
2025-09-07T09:09:09.4210337Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4210525Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4210588Z     return mod(**inputs)
2025-09-07T09:09:09.4210834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4210914Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4211155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4211211Z     outputs = block(
2025-09-07T09:09:09.4211436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4211507Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4211790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4211859Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4212112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4212192Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4212431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4212501Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4212787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4212975Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4213190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4213299Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4213304Z 
2025-09-07T09:09:09.4213383Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4213454Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4213531Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4213598Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4213691Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4213884Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4213942Z     return mod(**inputs)
2025-09-07T09:09:09.4214203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4214278Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4214522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4214585Z     outputs = block(
2025-09-07T09:09:09.4214802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4214879Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4215118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4215181Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4215422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4215497Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4215740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4215799Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4216045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4216350Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4216645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4216771Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4216775Z 
2025-09-07T09:09:09.4216869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4217063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4217122Z     return mod(**inputs)
2025-09-07T09:09:09.4217412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4217486Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4217766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4217833Z     outputs = block(
2025-09-07T09:09:09.4218048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4218121Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4218358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4218419Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4218662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4218792Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4219044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4219107Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4219359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4219452Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4219742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4219847Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4219851Z 
2025-09-07T09:09:09.4219942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4220136Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4220194Z     return mod(**inputs)
2025-09-07T09:09:09.4220440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4220520Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4220761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4220823Z     outputs = block(
2025-09-07T09:09:09.4221036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4221106Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4221349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4221408Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4221658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4221735Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4221974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4222036Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4222322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4222437Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4222651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4222762Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4222765Z 
2025-09-07T09:09:09.4222857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4223046Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4223107Z     return mod(**inputs)
2025-09-07T09:09:09.4223353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4223434Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4223706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4223767Z     outputs = block(
2025-09-07T09:09:09.4223989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4224058Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4224302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4224362Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4224647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4224744Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4224985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4225060Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4225276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4225389Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4225393Z 
2025-09-07T09:09:09.4225481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4225668Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4225735Z     return mod(**inputs)
2025-09-07T09:09:09.4225984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4226062Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4226300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4226354Z     outputs = block(
2025-09-07T09:09:09.4226577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4226641Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4226880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4226937Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4227183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4227274Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4227515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4227592Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4227800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4228020Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4228053Z 
2025-09-07T09:09:09.4228143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4228329Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4228395Z     return mod(**inputs)
2025-09-07T09:09:09.4228646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4228721Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4228966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4229025Z     outputs = block(
2025-09-07T09:09:09.4229238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4229304Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4229580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4229642Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4229890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4229977Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4230218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4230343Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4230558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4230669Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4230673Z 
2025-09-07T09:09:09.4230763Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4230957Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4231017Z     return mod(**inputs)
2025-09-07T09:09:09.4231259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4231339Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4231582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4231639Z     outputs = block(
2025-09-07T09:09:09.4231851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4231918Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4232158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4232219Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4232466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:09.4232566Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:09.4232569Z 
2025-09-07T09:09:09.4232657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4232845Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4232900Z     return mod(**inputs)
2025-09-07T09:09:09.4233154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4233225Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4233471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4233526Z     outputs = block(
2025-09-07T09:09:09.4233778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4233886Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4234130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4234194Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4234440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4234517Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4234769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4234829Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4235074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:09.4235291Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:09.4235506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4235617Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4235620Z 
2025-09-07T09:09:09.4235696Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4235773Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4235843Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4235915Z cudagraph partition due to non gpu ops
2025-09-07T09:09:09.4236046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4236236Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4236303Z     return mod(**inputs)
2025-09-07T09:09:09.4236549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4236626Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4236868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4236923Z     outputs = block(
2025-09-07T09:09:09.4237138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4237205Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4237449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4237509Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4237748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4237829Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4238065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4238132Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4238372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4238461Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4238763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:09.4238880Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:09.4238884Z 
2025-09-07T09:09:09.4238982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4239167Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4239232Z     return mod(**inputs)
2025-09-07T09:09:09.4239478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4239618Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4239868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4239924Z     outputs = block(
2025-09-07T09:09:09.4240141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4240207Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4240442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4240516Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4240759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4240837Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4241121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4241190Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4241432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:09.4241518Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:09.4241807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:09.4241951Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:09.4241955Z 
2025-09-07T09:09:09.4242050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4242237Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4242294Z     return mod(**inputs)
2025-09-07T09:09:09.4242546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4242618Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4242864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4242919Z     outputs = block(
2025-09-07T09:09:09.4243133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4243202Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4243439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4243502Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4243739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:09.4243819Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:09.4244053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4244112Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4244357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:09.4244427Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:09.4244642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4244744Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4244748Z 
2025-09-07T09:09:09.4244836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4245024Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4245081Z     return mod(**inputs)
2025-09-07T09:09:09.4245373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4245481Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4245736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4245791Z     outputs = block(
2025-09-07T09:09:09.4246004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4246081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4246319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4246391Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4246630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4246765Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4247017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:09.4247089Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:09.4247307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4253694Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4253706Z 
2025-09-07T09:09:09.4253843Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4254147Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4254213Z     return mod(**inputs)
2025-09-07T09:09:09.4254504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4254592Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4254868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4254935Z     outputs = block(
2025-09-07T09:09:09.4255162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4255239Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4255492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4255557Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4255809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4255908Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4256159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:09.4256238Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:09.4256451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:09.4256633Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:09.4256637Z 
2025-09-07T09:09:09.4256737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4256942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4257001Z     return mod(**inputs)
2025-09-07T09:09:09.4257255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:09.4257335Z     transformer_outputs = self.transformer(
2025-09-07T09:09:09.4257672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:09.4257781Z     outputs = block(
2025-09-07T09:09:09.4258034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:09.4258104Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:09.4258350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:09.4258409Z     return func(*args, **kwargs)
2025-09-07T09:09:09.4258653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:09.4258747Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:09.4258992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:09.4259073Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:09.4259320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:09.4259446Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:09.4259449Z 
2025-09-07T09:09:09.4259546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:09.4259747Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:09.4259803Z     return mod(**inputs)
2025-09-07T09:09:09.4260053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward
2025-09-07T09:09:09.4260203Z     logits = self.lm_head(hidden_states[:, slice_indices, :])
2025-09-07T09:09:09.4260207Z 
2025-09-07T09:09:23.7741661Z 
2025-09-07T09:09:23.9410924Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:09:24.1025661Z running benchmark:   7% 2/30 [00:00<00:02, 11.99it/s]
2025-09-07T09:09:24.2656871Z running benchmark:  13% 4/30 [00:00<00:02, 12.22it/s]
2025-09-07T09:09:24.4269643Z running benchmark:  20% 6/30 [00:00<00:01, 12.24it/s]
2025-09-07T09:09:24.5887599Z running benchmark:  27% 8/30 [00:00<00:01, 12.30it/s]
2025-09-07T09:09:24.7528651Z running benchmark:  33% 10/30 [00:00<00:01, 12.32it/s]
2025-09-07T09:09:24.9149153Z running benchmark:  40% 12/30 [00:00<00:01, 12.28it/s]
2025-09-07T09:09:25.0768332Z running benchmark:  47% 14/30 [00:01<00:01, 12.30it/s]
2025-09-07T09:09:25.2392163Z running benchmark:  53% 16/30 [00:01<00:01, 12.31it/s]
2025-09-07T09:09:25.3998801Z running benchmark:  60% 18/30 [00:01<00:00, 12.32it/s]
2025-09-07T09:09:25.5624156Z running benchmark:  67% 20/30 [00:01<00:00, 12.36it/s]
2025-09-07T09:09:25.7258720Z running benchmark:  73% 22/30 [00:01<00:00, 12.34it/s]
2025-09-07T09:09:25.8877999Z running benchmark:  80% 24/30 [00:01<00:00, 12.31it/s]
2025-09-07T09:09:26.0488993Z running benchmark:  87% 26/30 [00:02<00:00, 12.32it/s]
2025-09-07T09:09:26.2128811Z running benchmark:  93% 28/30 [00:02<00:00, 12.35it/s]
2025-09-07T09:09:26.2130170Z running benchmark: 100% 30/30 [00:02<00:00, 12.30it/s]
2025-09-07T09:09:26.2130471Z running benchmark: 100% 30/30 [00:02<00:00, 12.30it/s]
2025-09-07T09:09:26.3045518Z 1.328x
2025-09-07T09:09:26.3050556Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:09:28.8468527Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:09:28.8469420Z   import pynvml  # type: ignore[import]
2025-09-07T09:09:30.8585164Z 
2025-09-07T09:09:42.9709373Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:09:42.9710129Z loading model: 0it [00:12, ?it/s]
2025-09-07T09:09:43.0234222Z cpu  eval  hf_GPT2_large                      
2025-09-07T09:09:57.0574700Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0575448Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0575666Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0575986Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0576174Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0576370Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0576553Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0576751Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0576951Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0577144Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0577333Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0577535Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0577801Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0578001Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0578185Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0578380Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0578572Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0578855Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0579048Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0579239Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0579431Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0579624Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0579834Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0580024Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0580209Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0580404Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0580716Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0580910Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0581096Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0581289Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0581486Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0581680Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0581886Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0582088Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0582282Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0582471Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0582698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0583089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0583438Z     return mod(**inputs)
2025-09-07T09:09:57.0583855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0584265Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0584662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0585046Z     outputs = block(
2025-09-07T09:09:57.0585376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0585740Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0586120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0586502Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0586903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0587302Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0587695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0588070Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0588449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0589006Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0589513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0589926Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0590121Z 
2025-09-07T09:09:57.0590200Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0590409Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0590608Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0590797Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0591015Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0591387Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0591726Z     return mod(**inputs)
2025-09-07T09:09:57.0592114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0592551Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0592948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0593332Z     outputs = block(
2025-09-07T09:09:57.0593659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0594022Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0594396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0594820Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0595186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0595583Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0595975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0596343Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0596705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0597103Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0597558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0598057Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0598241Z 
2025-09-07T09:09:57.0598347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0598709Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0599045Z     return mod(**inputs)
2025-09-07T09:09:57.0599413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0599820Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0600225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0600604Z     outputs = block(
2025-09-07T09:09:57.0600966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0601340Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0601723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0602098Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0602469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0602872Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0603440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0603886Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0604431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0604853Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0605296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0605761Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0605929Z 
2025-09-07T09:09:57.0606032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0606395Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0606724Z     return mod(**inputs)
2025-09-07T09:09:57.0607144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0607553Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0607987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0608372Z     outputs = block(
2025-09-07T09:09:57.0608696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0609051Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0609480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0609866Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0610242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0610643Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0611030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0611405Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0611768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0612159Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0612507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0612928Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0613111Z 
2025-09-07T09:09:57.0613216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0613582Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0613929Z     return mod(**inputs)
2025-09-07T09:09:57.0614300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0614730Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0615124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0615498Z     outputs = block(
2025-09-07T09:09:57.0615812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0616177Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0616558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0616973Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0617338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0617877Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0618291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0618730Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0619077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0619484Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0619656Z 
2025-09-07T09:09:57.0619760Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0620134Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0620460Z     return mod(**inputs)
2025-09-07T09:09:57.0620832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0621222Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0621685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0622061Z     outputs = block(
2025-09-07T09:09:57.0622382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0622743Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0623112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0623481Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0623883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0624294Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0624697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0625075Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0625429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0625885Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0626125Z 
2025-09-07T09:09:57.0626237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0626596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0626919Z     return mod(**inputs)
2025-09-07T09:09:57.0627285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0627687Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0628068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0628443Z     outputs = block(
2025-09-07T09:09:57.0628762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0629129Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0629511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0629889Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0630242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0630654Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0631058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0631452Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0631816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0632299Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0632476Z 
2025-09-07T09:09:57.0632577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0632941Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0633264Z     return mod(**inputs)
2025-09-07T09:09:57.0633635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0634033Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0634421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0634797Z     outputs = block(
2025-09-07T09:09:57.0635123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0635513Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0635896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0636269Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0636636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0637033Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0637416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0637818Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0638178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0638675Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0639137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0639550Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0639725Z 
2025-09-07T09:09:57.0639810Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0640019Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0640228Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0640415Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0640638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0641012Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0641332Z     return mod(**inputs)
2025-09-07T09:09:57.0641680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0642070Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0642460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0642832Z     outputs = block(
2025-09-07T09:09:57.0643148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0643502Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0643883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0644252Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0644609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0644998Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0645384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0645817Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0646227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0646647Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0647095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0647591Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0647780Z 
2025-09-07T09:09:57.0647886Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0648253Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0648575Z     return mod(**inputs)
2025-09-07T09:09:57.0648930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0649380Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0649772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0650150Z     outputs = block(
2025-09-07T09:09:57.0650473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0650839Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0651223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0651601Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0652001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0652403Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0652782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0653153Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0653520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0653919Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0654359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0654828Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0654994Z 
2025-09-07T09:09:57.0655101Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0655456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0655781Z     return mod(**inputs)
2025-09-07T09:09:57.0656149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0656552Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0656937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0657312Z     outputs = block(
2025-09-07T09:09:57.0657674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0658027Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0658403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0658776Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0659141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0659530Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0659914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0660374Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0660729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0661120Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0661467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0661870Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0662047Z 
2025-09-07T09:09:57.0662154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0662509Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0662832Z     return mod(**inputs)
2025-09-07T09:09:57.0663180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0663624Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0664014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0664382Z     outputs = block(
2025-09-07T09:09:57.0664694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0665046Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0665423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0667053Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0667436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0667845Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0668251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0668638Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0668999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0669392Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0669558Z 
2025-09-07T09:09:57.0669657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0670014Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0670344Z     return mod(**inputs)
2025-09-07T09:09:57.0670703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0671092Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0671484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0671857Z     outputs = block(
2025-09-07T09:09:57.0672177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0672539Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0672909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0673300Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0673669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0674081Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0674486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0674862Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0675275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0675794Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0676027Z 
2025-09-07T09:09:57.0676133Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0676486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0676800Z     return mod(**inputs)
2025-09-07T09:09:57.0677168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0677556Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0677933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0678295Z     outputs = block(
2025-09-07T09:09:57.0678650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0679023Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0679395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0679778Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0680150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0680575Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0681024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0681432Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0681797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0682203Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0682386Z 
2025-09-07T09:09:57.0682489Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0682856Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0683185Z     return mod(**inputs)
2025-09-07T09:09:57.0683542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0683946Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0684332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0684717Z     outputs = block(
2025-09-07T09:09:57.0685052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0685415Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0685799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0686181Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0686547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0686980Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0687362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0687734Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0688100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0688599Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0689061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0689522Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0689739Z 
2025-09-07T09:09:57.0689823Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0690033Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0690231Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0690418Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0690645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0691012Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0691336Z     return mod(**inputs)
2025-09-07T09:09:57.0691689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0692084Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0692516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0692898Z     outputs = block(
2025-09-07T09:09:57.0693213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0693577Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0693954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0694327Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0694688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0695124Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0695512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0695877Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0696239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0696643Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0697088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0697580Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0697819Z 
2025-09-07T09:09:57.0697916Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0698269Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0698593Z     return mod(**inputs)
2025-09-07T09:09:57.0698938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0699334Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0699709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0700080Z     outputs = block(
2025-09-07T09:09:57.0700403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0700758Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0701130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0701496Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0701857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0702237Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0702617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0702985Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0703568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0704053Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0704497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0704963Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0705129Z 
2025-09-07T09:09:57.0705231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0705596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0705918Z     return mod(**inputs)
2025-09-07T09:09:57.0706269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0706668Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0707113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0707494Z     outputs = block(
2025-09-07T09:09:57.0707807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0708170Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0708548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0708922Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0709337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0709734Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0710122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0710501Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0710869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0711262Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0711614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0712008Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0712187Z 
2025-09-07T09:09:57.0712288Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0712648Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0712972Z     return mod(**inputs)
2025-09-07T09:09:57.0713341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0713739Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0714134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0714514Z     outputs = block(
2025-09-07T09:09:57.0714819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0715176Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0715551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0715919Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0716277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0716683Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0717082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0717541Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0717927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0718315Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0718489Z 
2025-09-07T09:09:57.0718590Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0718943Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0719265Z     return mod(**inputs)
2025-09-07T09:09:57.0719615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0720010Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0720392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0720759Z     outputs = block(
2025-09-07T09:09:57.0721111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0721468Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0721844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0722224Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0722584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0722989Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0723419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0723803Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0724146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0724605Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0724840Z 
2025-09-07T09:09:57.0724942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0725286Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0725602Z     return mod(**inputs)
2025-09-07T09:09:57.0725954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0726344Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0726719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0727090Z     outputs = block(
2025-09-07T09:09:57.0727398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0727754Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0728124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0728490Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0728853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0729254Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0729652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0730041Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0730389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0730782Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0731019Z 
2025-09-07T09:09:57.0731120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0731511Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0731823Z     return mod(**inputs)
2025-09-07T09:09:57.0732172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0732564Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0732941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0733306Z     outputs = block(
2025-09-07T09:09:57.0733611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0733963Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0734333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0734746Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0735105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.0735511Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.0735674Z 
2025-09-07T09:09:57.0735771Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0736122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0736438Z     return mod(**inputs)
2025-09-07T09:09:57.0736820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0737211Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0737588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0738006Z     outputs = block(
2025-09-07T09:09:57.0738319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0738673Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0739040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0739409Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0739764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0740145Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0740525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0740898Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0741263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0741762Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0742222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0742630Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0742808Z 
2025-09-07T09:09:57.0742885Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0743091Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0743288Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0743478Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0743696Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0744045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0744360Z     return mod(**inputs)
2025-09-07T09:09:57.0744706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0745181Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0745556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0745927Z     outputs = block(
2025-09-07T09:09:57.0746237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0746585Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0746957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0747329Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0747689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0748068Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0748500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0748877Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0749233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0749624Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0750064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0750585Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0750775Z 
2025-09-07T09:09:57.0750869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0751218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0751534Z     return mod(**inputs)
2025-09-07T09:09:57.0751883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0752281Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0752660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0753026Z     outputs = block(
2025-09-07T09:09:57.0753332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0753685Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0754055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0754424Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0754779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0755165Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0755542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0755909Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0756263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0756649Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0757093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0757553Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0757716Z 
2025-09-07T09:09:57.0757812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0758158Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0758519Z     return mod(**inputs)
2025-09-07T09:09:57.0758877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0759314Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0759703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0760068Z     outputs = block(
2025-09-07T09:09:57.0760370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0760725Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0761097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0761472Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0761822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0762253Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0762637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0763004Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0763358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0763739Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0764086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0764516Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0764690Z 
2025-09-07T09:09:57.0764798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0765153Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0765470Z     return mod(**inputs)
2025-09-07T09:09:57.0765845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0766245Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0766627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0766986Z     outputs = block(
2025-09-07T09:09:57.0767300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0767658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0768058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0768428Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0768781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0769188Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0769590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0769972Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0770317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0770702Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0770873Z 
2025-09-07T09:09:57.0770968Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0771334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0771664Z     return mod(**inputs)
2025-09-07T09:09:57.0772029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0772481Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0772912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0773288Z     outputs = block(
2025-09-07T09:09:57.0773613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0773969Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0774344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0774733Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0775099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0775499Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0775943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0776336Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0792302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0792867Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0793114Z 
2025-09-07T09:09:57.0793226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0793608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0794025Z     return mod(**inputs)
2025-09-07T09:09:57.0794408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0794808Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0795199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0795576Z     outputs = block(
2025-09-07T09:09:57.0795890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0796246Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0796614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0796981Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0797342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0797745Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0798140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0798524Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0798875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0799278Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0799449Z 
2025-09-07T09:09:57.0799556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0799909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0800246Z     return mod(**inputs)
2025-09-07T09:09:57.0800620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0801022Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0801409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0801773Z     outputs = block(
2025-09-07T09:09:57.0802090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0802589Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0802964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0803538Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0803909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0804305Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0804694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0805065Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0805443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0806024Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0806502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0806900Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0807070Z 
2025-09-07T09:09:57.0807148Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0807349Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0807537Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0807724Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0807940Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0808359Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0808681Z     return mod(**inputs)
2025-09-07T09:09:57.0809038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0809432Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0809808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0810170Z     outputs = block(
2025-09-07T09:09:57.0810483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0810835Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0811201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0811564Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0811917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0812298Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0812679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0813044Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0813405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0813802Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0814247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0814727Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0814908Z 
2025-09-07T09:09:57.0815011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0815361Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0815676Z     return mod(**inputs)
2025-09-07T09:09:57.0816028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0816513Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0816941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0817302Z     outputs = block(
2025-09-07T09:09:57.0817684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0818039Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0818409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0818771Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0819128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0819514Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0819929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0820293Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0820642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0821026Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0821463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0821912Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0822072Z 
2025-09-07T09:09:57.0822216Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0822562Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0822874Z     return mod(**inputs)
2025-09-07T09:09:57.0823228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0823619Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0823994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0824354Z     outputs = block(
2025-09-07T09:09:57.0824665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0825017Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0825383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0825747Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0826102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0826481Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0826853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0827209Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0827552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0827928Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0828275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0828660Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0828833Z 
2025-09-07T09:09:57.0828934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0829286Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0829599Z     return mod(**inputs)
2025-09-07T09:09:57.0829949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0830420Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0830802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0831166Z     outputs = block(
2025-09-07T09:09:57.0831486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0831841Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0832212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0832580Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0832941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0833345Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0833781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0834167Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0834509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0834896Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0835064Z 
2025-09-07T09:09:57.0835166Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0835553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0835868Z     return mod(**inputs)
2025-09-07T09:09:57.0836215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0836603Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0836980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0837339Z     outputs = block(
2025-09-07T09:09:57.0837647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0837997Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0838360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0838716Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0839068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0839462Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0839849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0840223Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0840554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0840998Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0841229Z 
2025-09-07T09:09:57.0841326Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0841668Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0841976Z     return mod(**inputs)
2025-09-07T09:09:57.0842316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0842696Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0843061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0843474Z     outputs = block(
2025-09-07T09:09:57.0843788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0844196Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0844562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0844929Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0845279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0845669Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0846059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0846439Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0846788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0847216Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0847387Z 
2025-09-07T09:09:57.0847482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0847820Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0848127Z     return mod(**inputs)
2025-09-07T09:09:57.0848483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0848870Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0849302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0849682Z     outputs = block(
2025-09-07T09:09:57.0850003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0850363Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0850731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0851100Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0851457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.0851865Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.0852023Z 
2025-09-07T09:09:57.0852125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0852473Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0852783Z     return mod(**inputs)
2025-09-07T09:09:57.0853132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0853519Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0853896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0854258Z     outputs = block(
2025-09-07T09:09:57.0854560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0854909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0855269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0855627Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0855979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0856363Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0856736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0857144Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0857497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0858093Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0858550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0858939Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0859108Z 
2025-09-07T09:09:57.0859182Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0859400Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0859595Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0859781Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0859997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0860831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0861172Z     return mod(**inputs)
2025-09-07T09:09:57.0861524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0861935Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0862302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0862686Z     outputs = block(
2025-09-07T09:09:57.0863013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0863405Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0863768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0864131Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0864508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0864898Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0865270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0865679Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0866037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0866424Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0866879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0867353Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0867548Z 
2025-09-07T09:09:57.0867652Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0867995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0868312Z     return mod(**inputs)
2025-09-07T09:09:57.0868684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0869066Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0869436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0869796Z     outputs = block(
2025-09-07T09:09:57.0870107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0870481Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0870839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0871218Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0871616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0872043Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0872435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0872792Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0873148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0873538Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0873998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0874443Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0874605Z 
2025-09-07T09:09:57.0874698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0875098Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0875418Z     return mod(**inputs)
2025-09-07T09:09:57.0875786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0876165Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0876553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0876918Z     outputs = block(
2025-09-07T09:09:57.0877274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0877639Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0878024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0878410Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0878786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0879180Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0879551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0879913Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0880263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0880649Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0880995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0881386Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0881563Z 
2025-09-07T09:09:57.0881660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0882010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0882345Z     return mod(**inputs)
2025-09-07T09:09:57.0882699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0883085Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0883460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0883831Z     outputs = block(
2025-09-07T09:09:57.0884145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0884508Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0884880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0885317Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0885678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0886138Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0886529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0886906Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0887247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0887653Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0887825Z 
2025-09-07T09:09:57.0887925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0888271Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0888615Z     return mod(**inputs)
2025-09-07T09:09:57.0889057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0889453Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0889821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0890187Z     outputs = block(
2025-09-07T09:09:57.0890514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0890876Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0891286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0891647Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0892012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0892411Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0892800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0893168Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0893505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0893944Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0894175Z 
2025-09-07T09:09:57.0894269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0894619Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0894925Z     return mod(**inputs)
2025-09-07T09:09:57.0895268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0895649Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0896035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0896410Z     outputs = block(
2025-09-07T09:09:57.0896707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0897050Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0897408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0897834Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0898186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0898607Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0898999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0899494Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0899850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0900244Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0900418Z 
2025-09-07T09:09:57.0900513Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0900863Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0901174Z     return mod(**inputs)
2025-09-07T09:09:57.0901539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0901925Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0902321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0902713Z     outputs = block(
2025-09-07T09:09:57.0903202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0903568Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0903961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0904333Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0904707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0905181Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0905552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0905934Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0906289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0906775Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0907243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0907636Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0907803Z 
2025-09-07T09:09:57.0907878Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0908085Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0908273Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0908482Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0908694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0909067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0909384Z     return mod(**inputs)
2025-09-07T09:09:57.0909731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0910112Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0910481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0910899Z     outputs = block(
2025-09-07T09:09:57.0911204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0911569Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0911938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0912308Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0912660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0913142Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0913575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0913943Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0914316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0914708Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0915145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0915621Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0915832Z 
2025-09-07T09:09:57.0915929Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0916276Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0916613Z     return mod(**inputs)
2025-09-07T09:09:57.0917006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0917399Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0917772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0918154Z     outputs = block(
2025-09-07T09:09:57.0918463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0918809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0919228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0919602Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0919972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0920365Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0920744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0921104Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0921459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0921850Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0922296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0922751Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0922912Z 
2025-09-07T09:09:57.0923009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0923364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0923688Z     return mod(**inputs)
2025-09-07T09:09:57.0924034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0924421Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0924790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0925150Z     outputs = block(
2025-09-07T09:09:57.0925452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0925800Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0926163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0926526Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0926887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0927340Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0927716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0928078Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0928424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0928804Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0929143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0929528Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0929696Z 
2025-09-07T09:09:57.0929792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0930130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0930477Z     return mod(**inputs)
2025-09-07T09:09:57.0930825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0931213Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0931578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0931938Z     outputs = block(
2025-09-07T09:09:57.0932238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0932622Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0932987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0933348Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0933692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0934091Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0934490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0934861Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0935208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0935584Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0935749Z 
2025-09-07T09:09:57.0935843Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0936183Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0936497Z     return mod(**inputs)
2025-09-07T09:09:57.0936837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0937219Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0937592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0937989Z     outputs = block(
2025-09-07T09:09:57.0938290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0938642Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0939007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0939365Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0939721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0940117Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0940561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0940975Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0941309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0941756Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0941986Z 
2025-09-07T09:09:57.0942086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0942427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0942742Z     return mod(**inputs)
2025-09-07T09:09:57.0943081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0943471Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0943879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0944253Z     outputs = block(
2025-09-07T09:09:57.0944563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0944919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0945279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0945634Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0946049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0946458Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0946854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0947252Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0947595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0947985Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0948158Z 
2025-09-07T09:09:57.0948251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0948586Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0948889Z     return mod(**inputs)
2025-09-07T09:09:57.0949244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0949626Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0950002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0950370Z     outputs = block(
2025-09-07T09:09:57.0950672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0951030Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0951390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0951764Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0952109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.0952505Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.0952660Z 
2025-09-07T09:09:57.0952752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0953086Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0953393Z     return mod(**inputs)
2025-09-07T09:09:57.0953727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0954183Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0954557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0954937Z     outputs = block(
2025-09-07T09:09:57.0955270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0955617Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0955977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0956334Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0956699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0957088Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0957529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0957902Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0958285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0958769Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0959221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0959642Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0959835Z 
2025-09-07T09:09:57.0959910Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0960108Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0960324Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0960529Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0960755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0961108Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0961446Z     return mod(**inputs)
2025-09-07T09:09:57.0961796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0962184Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0962584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0962966Z     outputs = block(
2025-09-07T09:09:57.0963278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0963627Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0963993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0964060Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0964304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0964383Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0964665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0964729Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0964971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0965054Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0965349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0965491Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0965536Z 
2025-09-07T09:09:57.0965634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0965864Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0965924Z     return mod(**inputs)
2025-09-07T09:09:57.0966192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0966273Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0966513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0966577Z     outputs = block(
2025-09-07T09:09:57.0966793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0966876Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0967147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0967213Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0967455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0967529Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0967783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0967845Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0968116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0968204Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0968495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0968643Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0968647Z 
2025-09-07T09:09:57.0968742Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0968933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0968990Z     return mod(**inputs)
2025-09-07T09:09:57.0969254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0969332Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0969574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0969633Z     outputs = block(
2025-09-07T09:09:57.0969844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0969911Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0970151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0970226Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0970476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0970551Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0970783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0970864Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0971107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0971182Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0971393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0971550Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0971584Z 
2025-09-07T09:09:57.0971679Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0971865Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0971927Z     return mod(**inputs)
2025-09-07T09:09:57.0972179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0972257Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0972498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0972555Z     outputs = block(
2025-09-07T09:09:57.0972789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0972859Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0973130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0973192Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0973433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0973534Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0973775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0973858Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0974100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0974208Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0974212Z 
2025-09-07T09:09:57.0974312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0974500Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0974562Z     return mod(**inputs)
2025-09-07T09:09:57.0974806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0974895Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0975138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0975196Z     outputs = block(
2025-09-07T09:09:57.0975414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0975481Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0975733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0975794Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0976036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0976132Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0976373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0976450Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0976658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0976833Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0976836Z 
2025-09-07T09:09:57.0977279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0977491Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0977559Z     return mod(**inputs)
2025-09-07T09:09:57.0977904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0978030Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0978284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0978344Z     outputs = block(
2025-09-07T09:09:57.0978562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0978633Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0978875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0978941Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0979204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0979350Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0979596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.0979678Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.0979893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0980031Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0980034Z 
2025-09-07T09:09:57.0980131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0980354Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0980427Z     return mod(**inputs)
2025-09-07T09:09:57.0980681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0980764Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0981007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0981089Z     outputs = block(
2025-09-07T09:09:57.0981305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0981374Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0981627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0981696Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0981939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0982017Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0982250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0982318Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0982558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.0982743Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.0982956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0983067Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0983070Z 
2025-09-07T09:09:57.0983145Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0983213Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0983310Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0983376Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.0983497Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0983722Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0983812Z     return mod(**inputs)
2025-09-07T09:09:57.0984085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0984159Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0984405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0984460Z     outputs = block(
2025-09-07T09:09:57.0984695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0984771Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0985005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0985071Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0985345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0985425Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0985660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0985738Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0986000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0986091Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0986420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.0986543Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.0986547Z 
2025-09-07T09:09:57.0986638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0986827Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0986888Z     return mod(**inputs)
2025-09-07T09:09:57.0987152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0987224Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0987464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0987522Z     outputs = block(
2025-09-07T09:09:57.0987735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0987807Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0988045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0988108Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0988350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0988426Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0988667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0988726Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0988972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.0989057Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.0989346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.0989452Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.0989456Z 
2025-09-07T09:09:57.0989592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0989824Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0989884Z     return mod(**inputs)
2025-09-07T09:09:57.0990132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0990216Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0990464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0990527Z     outputs = block(
2025-09-07T09:09:57.0990742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0990821Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0991062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0991165Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0991418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.0991494Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.0991737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0991800Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0992041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.0992159Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.0992376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0992493Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0992497Z 
2025-09-07T09:09:57.0992592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0992791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0992852Z     return mod(**inputs)
2025-09-07T09:09:57.0993100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0993185Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0993429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0993491Z     outputs = block(
2025-09-07T09:09:57.0993706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0993778Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0994019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0994084Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0994335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0994432Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0994673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.0994754Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.0994967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.0995076Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.0995079Z 
2025-09-07T09:09:57.0995172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0995365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0995491Z     return mod(**inputs)
2025-09-07T09:09:57.0995743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0995869Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0996112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0996176Z     outputs = block(
2025-09-07T09:09:57.0996388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0996459Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0996703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0996765Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0997015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0997143Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.0997391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.0997472Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.0997681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.0997866Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.0997869Z 
2025-09-07T09:09:57.0998009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.0998209Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.0998269Z     return mod(**inputs)
2025-09-07T09:09:57.0998518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.0998604Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.0998846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.0998912Z     outputs = block(
2025-09-07T09:09:57.0999129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.0999197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.0999448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.0999515Z     return func(*args, **kwargs)
2025-09-07T09:09:57.0999763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.0999858Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1000112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1000198Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1000413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1000528Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1000531Z 
2025-09-07T09:09:57.1000624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1000817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1000878Z     return mod(**inputs)
2025-09-07T09:09:57.1001124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1001206Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1001450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1001600Z     outputs = block(
2025-09-07T09:09:57.1001814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1001884Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1002125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1002190Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1002435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1002535Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1002539Z 
2025-09-07T09:09:57.1002634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1002817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1002933Z     return mod(**inputs)
2025-09-07T09:09:57.1003338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1003413Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1003665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1003726Z     outputs = block(
2025-09-07T09:09:57.1003943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1004088Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1004337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1004409Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1004659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1004748Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1005001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1005070Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1005325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1005509Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1005731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1005845Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1005849Z 
2025-09-07T09:09:57.1005924Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1005996Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1006066Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1006140Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1006237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1006429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1006495Z     return mod(**inputs)
2025-09-07T09:09:57.1006744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1006823Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1007067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1007127Z     outputs = block(
2025-09-07T09:09:57.1007349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1007470Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1007714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1007820Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1008064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1008145Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1008381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1008446Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1008691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1008788Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1009126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1009255Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1009258Z 
2025-09-07T09:09:57.1009361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1009553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1009620Z     return mod(**inputs)
2025-09-07T09:09:57.1009870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1009946Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1010229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1010288Z     outputs = block(
2025-09-07T09:09:57.1010516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1010593Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1010843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1010909Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1011155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1011239Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1011477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1011547Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1011787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1011873Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1012171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1012280Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1012283Z 
2025-09-07T09:09:57.1012387Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1012575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1012641Z     return mod(**inputs)
2025-09-07T09:09:57.1012886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1012964Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1013212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1013271Z     outputs = block(
2025-09-07T09:09:57.1013495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1013603Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1013879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1013949Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1014191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1014276Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1014513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1014577Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1014826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1014902Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1015154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1015263Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1015267Z 
2025-09-07T09:09:57.1015368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1015555Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1015616Z     return mod(**inputs)
2025-09-07T09:09:57.1015868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1015972Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1016227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1016284Z     outputs = block(
2025-09-07T09:09:57.1016504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1016584Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1016817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1016886Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1017127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1017226Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1017474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1017544Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1017801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1017904Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1017910Z 
2025-09-07T09:09:57.1018007Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1018193Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1018253Z     return mod(**inputs)
2025-09-07T09:09:57.1018506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1018580Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1018831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1018887Z     outputs = block(
2025-09-07T09:09:57.1019101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1019179Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1019416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1019552Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1019797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1019894Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1020137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1020208Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1020430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1020603Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1020606Z 
2025-09-07T09:09:57.1020706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1020925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1020987Z     return mod(**inputs)
2025-09-07T09:09:57.1021241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1021315Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1021561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1021619Z     outputs = block(
2025-09-07T09:09:57.1021865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1021944Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1022182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1022254Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1022499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1022600Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1022840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1022922Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1023138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1023245Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1023248Z 
2025-09-07T09:09:57.1023349Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1023535Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1023595Z     return mod(**inputs)
2025-09-07T09:09:57.1023851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1023929Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1024175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1024231Z     outputs = block(
2025-09-07T09:09:57.1024451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1024521Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1024761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1024832Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1025073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1025208Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1025447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1025568Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1025816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1025998Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1026216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1026320Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1026324Z 
2025-09-07T09:09:57.1026405Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1026479Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1026551Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1026662Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1026756Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1026950Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1027012Z     return mod(**inputs)
2025-09-07T09:09:57.1027258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1027339Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1027581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1027677Z     outputs = block(
2025-09-07T09:09:57.1027900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1027972Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1028221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1028287Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1028537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1028614Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1028850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1028920Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1029163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1029260Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1029553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1029682Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1029686Z 
2025-09-07T09:09:57.1029778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1029961Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1030030Z     return mod(**inputs)
2025-09-07T09:09:57.1030274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1030355Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1030594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1030651Z     outputs = block(
2025-09-07T09:09:57.1030872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1030944Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1031231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1031324Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1031569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1031651Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1031888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1031959Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1032201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1032287Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1032573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1032705Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1032711Z 
2025-09-07T09:09:57.1032806Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1032989Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1033052Z     return mod(**inputs)
2025-09-07T09:09:57.1033299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1033370Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1033648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1033704Z     outputs = block(
2025-09-07T09:09:57.1033923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1033991Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1034230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1034296Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1034536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1034613Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1034849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1034913Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1035154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1035229Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1035442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1035549Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1035554Z 
2025-09-07T09:09:57.1035650Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1035831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1035887Z     return mod(**inputs)
2025-09-07T09:09:57.1036134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1036206Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1036454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1036508Z     outputs = block(
2025-09-07T09:09:57.1036722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1036790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1037063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1037156Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1037396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1037493Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1037734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1037805Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1038024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1038126Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1038130Z 
2025-09-07T09:09:57.1038226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1038441Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1038501Z     return mod(**inputs)
2025-09-07T09:09:57.1038753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1038825Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1039073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1039130Z     outputs = block(
2025-09-07T09:09:57.1039385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1039453Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1039689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1039761Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1040004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1040101Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1040341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1040411Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1040626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1040799Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1040803Z 
2025-09-07T09:09:57.1040899Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1041080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1041143Z     return mod(**inputs)
2025-09-07T09:09:57.1041390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1041462Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1041708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1041764Z     outputs = block(
2025-09-07T09:09:57.1041982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1042050Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1042282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1042350Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1042592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1042733Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1043006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1043085Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1043306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1043411Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1043415Z 
2025-09-07T09:09:57.1043514Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1043695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1043760Z     return mod(**inputs)
2025-09-07T09:09:57.1044004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1044114Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1044366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1053368Z     outputs = block(
2025-09-07T09:09:57.1053668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1053748Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1054022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1054191Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1054457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1054564Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1054569Z 
2025-09-07T09:09:57.1054680Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1054893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1054957Z     return mod(**inputs)
2025-09-07T09:09:57.1055216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1055298Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1055542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1055602Z     outputs = block(
2025-09-07T09:09:57.1055825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1055897Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1056140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1056208Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1056455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1056536Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1056771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1056834Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1057073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1057260Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1057477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1057597Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1057734Z 
2025-09-07T09:09:57.1057810Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1057920Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1057992Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1058058Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1058158Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1058353Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1058411Z     return mod(**inputs)
2025-09-07T09:09:57.1058678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1058758Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1059030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1059090Z     outputs = block(
2025-09-07T09:09:57.1059366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1059444Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1059690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1059754Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1060001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1060080Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1060355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1060415Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1060659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1060753Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1061059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1061183Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1061187Z 
2025-09-07T09:09:57.1061285Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1061485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1061544Z     return mod(**inputs)
2025-09-07T09:09:57.1061798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1061872Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1062113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1062174Z     outputs = block(
2025-09-07T09:09:57.1062389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1062460Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1062695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1062756Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1062996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1063067Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1063305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1063364Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1063603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1063722Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1064048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1064153Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1064156Z 
2025-09-07T09:09:57.1064245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1064436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1064495Z     return mod(**inputs)
2025-09-07T09:09:57.1064745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1064821Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1065059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1065148Z     outputs = block(
2025-09-07T09:09:57.1065367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1065439Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1065675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1065736Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1065978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1066082Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1066320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1066377Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1066620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1066698Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1066913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1067026Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1067029Z 
2025-09-07T09:09:57.1067118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1067303Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1067365Z     return mod(**inputs)
2025-09-07T09:09:57.1067612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1067686Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1067923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1067983Z     outputs = block(
2025-09-07T09:09:57.1068197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1068264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1068505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1068562Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1068805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1068901Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1069137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1069212Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1069421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1069592Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1069595Z 
2025-09-07T09:09:57.1069684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1069870Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1069927Z     return mod(**inputs)
2025-09-07T09:09:57.1070174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1070248Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1070487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1070541Z     outputs = block(
2025-09-07T09:09:57.1070751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1070850Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1071089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1071147Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1071387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1071473Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1071745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1071819Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1072026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1072204Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1072210Z 
2025-09-07T09:09:57.1072304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1072491Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1072545Z     return mod(**inputs)
2025-09-07T09:09:57.1072788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1072858Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1073098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1073152Z     outputs = block(
2025-09-07T09:09:57.1073362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1073426Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1073664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1073726Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1073967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1074055Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1074294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1074371Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1074583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1074694Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1074697Z 
2025-09-07T09:09:57.1074788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1074979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1075098Z     return mod(**inputs)
2025-09-07T09:09:57.1075346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1075421Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1075661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1075716Z     outputs = block(
2025-09-07T09:09:57.1075931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1076000Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1076239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1076299Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1076593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1076676Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1076915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1076974Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1077211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1077426Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1077640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1077746Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1077750Z 
2025-09-07T09:09:57.1077823Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1077894Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1077964Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1078030Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1078122Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1078309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1078368Z     return mod(**inputs)
2025-09-07T09:09:57.1078614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1078686Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1078926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1078979Z     outputs = block(
2025-09-07T09:09:57.1079190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1079261Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1079494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1079555Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1079791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1079865Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1080097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1080156Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1080394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1080478Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1080977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1081125Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1081129Z 
2025-09-07T09:09:57.1081217Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1081404Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1081460Z     return mod(**inputs)
2025-09-07T09:09:57.1081709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1081780Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1082020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1082073Z     outputs = block(
2025-09-07T09:09:57.1082319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1082394Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1082626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1082689Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1082936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1083010Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1083276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1083334Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1083576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1083658Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1083945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1084047Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1084051Z 
2025-09-07T09:09:57.1084139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1084326Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1084381Z     return mod(**inputs)
2025-09-07T09:09:57.1084628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1084698Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1084937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1084991Z     outputs = block(
2025-09-07T09:09:57.1085206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1085276Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1085508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1085568Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1085808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1085881Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1086119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1086178Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1086416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1086527Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1086739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1086876Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1086879Z 
2025-09-07T09:09:57.1086965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1087149Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1087204Z     return mod(**inputs)
2025-09-07T09:09:57.1087447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1087519Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1087756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1087811Z     outputs = block(
2025-09-07T09:09:57.1088054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1088122Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1088359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1088416Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1088661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1088752Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1089022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1089090Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1089297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1089402Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1089407Z 
2025-09-07T09:09:57.1089493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1089676Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1089731Z     return mod(**inputs)
2025-09-07T09:09:57.1089973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1090041Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1090280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1090335Z     outputs = block(
2025-09-07T09:09:57.1090545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1090610Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1090849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1090908Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1091150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1091236Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1091479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1091545Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1091754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1091927Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1091930Z 
2025-09-07T09:09:57.1092055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1092240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1092327Z     return mod(**inputs)
2025-09-07T09:09:57.1092578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1092658Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1092906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1092962Z     outputs = block(
2025-09-07T09:09:57.1093184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1093258Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1093497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1093596Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1093845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1093939Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1094180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1094256Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1094469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1094615Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1094618Z 
2025-09-07T09:09:57.1094712Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1094905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1094964Z     return mod(**inputs)
2025-09-07T09:09:57.1095215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1095290Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1095533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1095589Z     outputs = block(
2025-09-07T09:09:57.1095803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1095878Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1096118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1096179Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1096421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1096521Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1096525Z 
2025-09-07T09:09:57.1096617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1096803Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1096858Z     return mod(**inputs)
2025-09-07T09:09:57.1097105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1097176Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1097421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1097475Z     outputs = block(
2025-09-07T09:09:57.1097725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1097795Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1098082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1098197Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1098438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1098518Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1098754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1098813Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1099054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1099236Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1099485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1099594Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1099597Z 
2025-09-07T09:09:57.1099674Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1099741Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1099807Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1099876Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1099964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1100147Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1100265Z     return mod(**inputs)
2025-09-07T09:09:57.1100516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1100591Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1100836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1100894Z     outputs = block(
2025-09-07T09:09:57.1101110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1101178Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1101417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1101476Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1101721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1101796Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1102030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1102091Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1102335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1102424Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1102714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1102838Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1102844Z 
2025-09-07T09:09:57.1102938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1103314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1103373Z     return mod(**inputs)
2025-09-07T09:09:57.1103618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1103696Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1104040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1104144Z     outputs = block(
2025-09-07T09:09:57.1104371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1104443Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1104686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1104747Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1104989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1105066Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1105301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1105367Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1105659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1105749Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1106041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1106142Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1106146Z 
2025-09-07T09:09:57.1106240Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1106467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1106526Z     return mod(**inputs)
2025-09-07T09:09:57.1106778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1106853Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1107104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1107159Z     outputs = block(
2025-09-07T09:09:57.1107378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1107450Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1107689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1107756Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1107996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1108078Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1108314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1108377Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1108615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1108687Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1108904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1109021Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1109025Z 
2025-09-07T09:09:57.1109127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1109320Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1109379Z     return mod(**inputs)
2025-09-07T09:09:57.1109630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1109745Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1109992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1110079Z     outputs = block(
2025-09-07T09:09:57.1110296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1110369Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1110605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1110670Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1110908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1111010Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1111247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1111353Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1111573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1111677Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1111680Z 
2025-09-07T09:09:57.1111774Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1111960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1112017Z     return mod(**inputs)
2025-09-07T09:09:57.1112299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1112372Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1112617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1112674Z     outputs = block(
2025-09-07T09:09:57.1112895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1112963Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1113201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1113264Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1113503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1113598Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1113834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1113902Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1114116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1114291Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1114295Z 
2025-09-07T09:09:57.1114389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1114572Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1114634Z     return mod(**inputs)
2025-09-07T09:09:57.1114877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1114953Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1115197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1115253Z     outputs = block(
2025-09-07T09:09:57.1115468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1115573Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1115841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1115904Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1116142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1116236Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1116479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1116556Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1116769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1116870Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1116875Z 
2025-09-07T09:09:57.1116997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1117185Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1117244Z     return mod(**inputs)
2025-09-07T09:09:57.1117488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1117559Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1117805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1117889Z     outputs = block(
2025-09-07T09:09:57.1118108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1118174Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1118412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1118480Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1118717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1118797Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1119037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1119102Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1119348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1119526Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1119741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1119846Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1119851Z 
2025-09-07T09:09:57.1119929Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1119995Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1120061Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1120133Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1120222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1120410Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1120466Z     return mod(**inputs)
2025-09-07T09:09:57.1120712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1120790Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1121031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1121139Z     outputs = block(
2025-09-07T09:09:57.1121352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1121451Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1121692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1121751Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1121994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1122072Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1122310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1122370Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1122608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1122733Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1123032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1123153Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1123157Z 
2025-09-07T09:09:57.1123247Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1123430Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1123491Z     return mod(**inputs)
2025-09-07T09:09:57.1123770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1123845Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1124085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1124141Z     outputs = block(
2025-09-07T09:09:57.1124361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1124428Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1124666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1124725Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1124968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1125042Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1125274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1125338Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1125579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1125663Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1125951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1126048Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1126052Z 
2025-09-07T09:09:57.1126145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1126328Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1126389Z     return mod(**inputs)
2025-09-07T09:09:57.1126636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1126711Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1126954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1127095Z     outputs = block(
2025-09-07T09:09:57.1127311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1127376Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1127614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1127671Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1127913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1127989Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1128222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1128282Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1128551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1128625Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1128836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1128937Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1128940Z 
2025-09-07T09:09:57.1129032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1129215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1129303Z     return mod(**inputs)
2025-09-07T09:09:57.1129550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1129622Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1129867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1129923Z     outputs = block(
2025-09-07T09:09:57.1130140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1130205Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1130437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1130501Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1130741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1130838Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1131077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1131143Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1131362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1131464Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1131467Z 
2025-09-07T09:09:57.1131557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1131738Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1131800Z     return mod(**inputs)
2025-09-07T09:09:57.1132044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1132118Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1132358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1132411Z     outputs = block(
2025-09-07T09:09:57.1132627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1132753Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1132987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1133050Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1133291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1133383Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1133623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1133694Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1133903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1134104Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1134110Z 
2025-09-07T09:09:57.1134203Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1134387Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1134445Z     return mod(**inputs)
2025-09-07T09:09:57.1134689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1134757Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1135031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1135087Z     outputs = block(
2025-09-07T09:09:57.1135302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1135366Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1135600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1135664Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1135904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1135992Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1136233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1136312Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1136527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1136627Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1136631Z 
2025-09-07T09:09:57.1136724Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1136909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1136970Z     return mod(**inputs)
2025-09-07T09:09:57.1137214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1137284Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1137527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1137580Z     outputs = block(
2025-09-07T09:09:57.1137876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1137946Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1138186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1138285Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1138528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1138664Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1138668Z 
2025-09-07T09:09:57.1138757Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1138940Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1138997Z     return mod(**inputs)
2025-09-07T09:09:57.1139242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1139315Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1139554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1139609Z     outputs = block(
2025-09-07T09:09:57.1139854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1139923Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1140167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1140227Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1140467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1140543Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1140828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1140888Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1141125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1141312Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1141524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1141630Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1141633Z 
2025-09-07T09:09:57.1141705Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1141773Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1141844Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1141911Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1142005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1142188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1142244Z     return mod(**inputs)
2025-09-07T09:09:57.1142490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1142564Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1142811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1142863Z     outputs = block(
2025-09-07T09:09:57.1143075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1143144Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1143381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1143446Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1143686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1143761Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1143995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1144117Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1144360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1144445Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1144740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1144855Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1144858Z 
2025-09-07T09:09:57.1144949Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1145131Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1145187Z     return mod(**inputs)
2025-09-07T09:09:57.1145464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1145537Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1145783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1145838Z     outputs = block(
2025-09-07T09:09:57.1146050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1146118Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1146382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1146443Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1146685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1146758Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1147001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1147061Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1147304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1147383Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1147666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1147762Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1147767Z 
2025-09-07T09:09:57.1147856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1148044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1148100Z     return mod(**inputs)
2025-09-07T09:09:57.1148354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1148429Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1148677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1148732Z     outputs = block(
2025-09-07T09:09:57.1148949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1149020Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1149260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1149322Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1149566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1149643Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1149929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1150020Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1150268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1150345Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1150559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1150669Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1150673Z 
2025-09-07T09:09:57.1150767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1150958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1151014Z     return mod(**inputs)
2025-09-07T09:09:57.1151294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1151371Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1151611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1151666Z     outputs = block(
2025-09-07T09:09:57.1151878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1151947Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1152222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1152285Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1152528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1152623Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1152870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1152940Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1153150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1153252Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1153255Z 
2025-09-07T09:09:57.1153346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1153536Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1153595Z     return mod(**inputs)
2025-09-07T09:09:57.1153841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1153915Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1154158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1154219Z     outputs = block(
2025-09-07T09:09:57.1154432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1154497Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1154737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1154797Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1155043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1155135Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1155379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1155488Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1155731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1155902Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1155905Z 
2025-09-07T09:09:57.1155999Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1156189Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1156246Z     return mod(**inputs)
2025-09-07T09:09:57.1156493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1156567Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1156807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1156900Z     outputs = block(
2025-09-07T09:09:57.1157115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1157189Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1157423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1157483Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1157724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1157842Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1158091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1158168Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1158381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1158494Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1158497Z 
2025-09-07T09:09:57.1158595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1158781Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1158835Z     return mod(**inputs)
2025-09-07T09:09:57.1159084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1159158Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1159400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1159461Z     outputs = block(
2025-09-07T09:09:57.1159672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1159747Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1159982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1160041Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1160289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1160366Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1160601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1160661Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1160898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1161082Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1161337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1161476Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1161479Z 
2025-09-07T09:09:57.1161551Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1161621Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1161689Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1161753Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1161850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1162038Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1162098Z     return mod(**inputs)
2025-09-07T09:09:57.1162342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1162413Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1162702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1162759Z     outputs = block(
2025-09-07T09:09:57.1162976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1163043Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1163278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1163343Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1163621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1163706Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1163943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1164009Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1164258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1164351Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1164654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1164775Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1164778Z 
2025-09-07T09:09:57.1164872Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1165058Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1165118Z     return mod(**inputs)
2025-09-07T09:09:57.1165370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1165445Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1165693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1165748Z     outputs = block(
2025-09-07T09:09:57.1165961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1166035Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1166272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1166340Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1166580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1166660Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1166898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1166994Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1167280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1167372Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1167677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1167784Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1167787Z 
2025-09-07T09:09:57.1167886Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1168089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1168148Z     return mod(**inputs)
2025-09-07T09:09:57.1168420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1168528Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1168784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1168847Z     outputs = block(
2025-09-07T09:09:57.1169066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1169140Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1169380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1169481Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1169729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1169809Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1170050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1170113Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1170361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1170435Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1170648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1170770Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1170774Z 
2025-09-07T09:09:57.1170873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1171068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1171128Z     return mod(**inputs)
2025-09-07T09:09:57.1171377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1171459Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1171701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1171762Z     outputs = block(
2025-09-07T09:09:57.1171975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1172049Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1172285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1172348Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1172591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1172686Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1172934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1173066Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1173279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1173387Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1173390Z 
2025-09-07T09:09:57.1173480Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1173671Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1173728Z     return mod(**inputs)
2025-09-07T09:09:57.1173977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1174049Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1174324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1174387Z     outputs = block(
2025-09-07T09:09:57.1174602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1174679Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1174915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1174975Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1175221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1175341Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1175594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1175663Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1175874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1176062Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1176066Z 
2025-09-07T09:09:57.1176157Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1176346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1176404Z     return mod(**inputs)
2025-09-07T09:09:57.1176652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1176722Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1176964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1177028Z     outputs = block(
2025-09-07T09:09:57.1177243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1177319Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1177556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1177660Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1177907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1178000Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1178249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1178325Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1178541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1178682Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1178716Z 
2025-09-07T09:09:57.1178808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1179002Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1179058Z     return mod(**inputs)
2025-09-07T09:09:57.1179309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1179380Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1179624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1179685Z     outputs = block(
2025-09-07T09:09:57.1179898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1179970Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1180241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1180306Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1180557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1180656Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1180659Z 
2025-09-07T09:09:57.1180757Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1180939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1181035Z     return mod(**inputs)
2025-09-07T09:09:57.1181283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1181355Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1181607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1181664Z     outputs = block(
2025-09-07T09:09:57.1181882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1181949Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1182185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1182252Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1182494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1182576Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1182814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1182874Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1183122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1183305Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1183525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1183629Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1183632Z 
2025-09-07T09:09:57.1183712Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1183781Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1183850Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1183923Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1184012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1184203Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1184301Z     return mod(**inputs)
2025-09-07T09:09:57.1184550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1184677Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1184941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1185003Z     outputs = block(
2025-09-07T09:09:57.1185223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1185293Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1185535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1185596Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1185842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1185952Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1186203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1186264Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1186505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1186602Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1186927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1187053Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1187056Z 
2025-09-07T09:09:57.1187146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1187329Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1187397Z     return mod(**inputs)
2025-09-07T09:09:57.1187644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1187721Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1187961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1188017Z     outputs = block(
2025-09-07T09:09:57.1188237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1188306Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1188548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1188606Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1188852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1188931Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1189164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1189231Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1189468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1189553Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1189845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1189942Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1189950Z 
2025-09-07T09:09:57.1190037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1190223Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1190359Z     return mod(**inputs)
2025-09-07T09:09:57.1190604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1190679Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1190921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1190973Z     outputs = block(
2025-09-07T09:09:57.1191191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1191260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1191503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1191561Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1191834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1191916Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1192150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1192214Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1192451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1192523Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1192774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1192881Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1192884Z 
2025-09-07T09:09:57.1192977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1193159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1193223Z     return mod(**inputs)
2025-09-07T09:09:57.1193466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1193537Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1193779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1193833Z     outputs = block(
2025-09-07T09:09:57.1194051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1194118Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1194351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1194413Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1194656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1194752Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1194992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1195060Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1195276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1195373Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1195376Z 
2025-09-07T09:09:57.1195469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1195648Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1195708Z     return mod(**inputs)
2025-09-07T09:09:57.1195952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1196085Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1196332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1196386Z     outputs = block(
2025-09-07T09:09:57.1196602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1196669Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1196905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1196970Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1197206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1197298Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1197572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1197647Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1197852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1198021Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1198024Z 
2025-09-07T09:09:57.1198116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1198330Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1198391Z     return mod(**inputs)
2025-09-07T09:09:57.1198638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1198710Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1198955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1199010Z     outputs = block(
2025-09-07T09:09:57.1199227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1199294Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1199526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1199590Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1199834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1199926Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1200167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1200248Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1200457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1200555Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1200559Z 
2025-09-07T09:09:57.1200649Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1200830Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1200889Z     return mod(**inputs)
2025-09-07T09:09:57.1201132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1201203Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1201444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1201531Z     outputs = block(
2025-09-07T09:09:57.1201752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1201849Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1202090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1202151Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1202391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1202470Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1202707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1202767Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1203006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1203385Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1203607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1203714Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1203718Z 
2025-09-07T09:09:57.1203795Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1203862Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1203932Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1204003Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1204140Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1204337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1204393Z     return mod(**inputs)
2025-09-07T09:09:57.1204647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1204729Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1204973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1205031Z     outputs = block(
2025-09-07T09:09:57.1205247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1205321Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1205564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1205633Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1205887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1205969Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1206208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1206269Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1206507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1206599Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1206896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1207019Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1207023Z 
2025-09-07T09:09:57.1207113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1207300Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1207358Z     return mod(**inputs)
2025-09-07T09:09:57.1207668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1207786Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1208024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1208083Z     outputs = block(
2025-09-07T09:09:57.1208297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1208364Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1208607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1208668Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1208916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1208990Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1209261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1209328Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1209570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1209653Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1209942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1210088Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1210091Z 
2025-09-07T09:09:57.1210184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1210369Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1210433Z     return mod(**inputs)
2025-09-07T09:09:57.1210682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1210761Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1211012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1211066Z     outputs = block(
2025-09-07T09:09:57.1211283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1211348Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1211588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1211647Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1211883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1211964Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1212201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1212261Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1212499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1212572Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1212781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1212886Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1212889Z 
2025-09-07T09:09:57.1212981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1213166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1213263Z     return mod(**inputs)
2025-09-07T09:09:57.1213514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1213619Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1213867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1213924Z     outputs = block(
2025-09-07T09:09:57.1214143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1214211Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1214456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1214516Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1214756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1214888Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1215134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1215208Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1215419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1215522Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1215525Z 
2025-09-07T09:09:57.1215620Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1215873Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1215934Z     return mod(**inputs)
2025-09-07T09:09:57.1216181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1216257Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1216503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1216561Z     outputs = block(
2025-09-07T09:09:57.1216779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1216844Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1217084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1217144Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1217386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1217487Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1217801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1217876Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1218087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1218259Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1218267Z 
2025-09-07T09:09:57.1218356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1218539Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1218607Z     return mod(**inputs)
2025-09-07T09:09:57.1218853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1218930Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1219170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1219270Z     outputs = block(
2025-09-07T09:09:57.1219524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1219588Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1219833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1219890Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1220127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1220222Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1220460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1220537Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1220783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1220885Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1220894Z 
2025-09-07T09:09:57.1220983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1221165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1221225Z     return mod(**inputs)
2025-09-07T09:09:57.1221470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1221577Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1221820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1221874Z     outputs = block(
2025-09-07T09:09:57.1222092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1222161Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1222402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1222461Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1222707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1222805Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1222808Z 
2025-09-07T09:09:57.1222902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1223094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1223149Z     return mod(**inputs)
2025-09-07T09:09:57.1223400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1223477Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1223722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1223779Z     outputs = block(
2025-09-07T09:09:57.1223996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1224065Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1224304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1224365Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1224606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1224681Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1224928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1225047Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1225287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1225473Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1225687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1225798Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1225801Z 
2025-09-07T09:09:57.1225875Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1225947Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1226012Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1226077Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1226169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1226390Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1226453Z     return mod(**inputs)
2025-09-07T09:09:57.1226701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1226774Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1227024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1227081Z     outputs = block(
2025-09-07T09:09:57.1227351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1227422Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1227661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1227732Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1227973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1228058Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1228291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1228352Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1228599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1228689Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1228990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1229110Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1229113Z 
2025-09-07T09:09:57.1229215Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1229405Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1229463Z     return mod(**inputs)
2025-09-07T09:09:57.1229712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1229781Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1230028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1230084Z     outputs = block(
2025-09-07T09:09:57.1230299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1230375Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1230611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1230722Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1230997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1231078Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1231312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1231370Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1231613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1231696Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1231991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1232090Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1232094Z 
2025-09-07T09:09:57.1232224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1232418Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1232475Z     return mod(**inputs)
2025-09-07T09:09:57.1232732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1232805Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1233050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1233145Z     outputs = block(
2025-09-07T09:09:57.1233363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1233438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1233675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1233745Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1233987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1234063Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1234305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1234367Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1234614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1234687Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1234900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1235012Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1235017Z 
2025-09-07T09:09:57.1235108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1235296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1235352Z     return mod(**inputs)
2025-09-07T09:09:57.1235599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1235670Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1235909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1235973Z     outputs = block(
2025-09-07T09:09:57.1236184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1236258Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1236495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1236590Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1236865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1236959Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1237203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1237272Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1237480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1237583Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1237586Z 
2025-09-07T09:09:57.1237675Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1237864Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1237954Z     return mod(**inputs)
2025-09-07T09:09:57.1238212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1238286Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1238528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1238591Z     outputs = block(
2025-09-07T09:09:57.1238804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1238909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1239151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1239213Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1239464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1239557Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1239806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1239875Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1240081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1240254Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1240258Z 
2025-09-07T09:09:57.1240346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1240531Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1240588Z     return mod(**inputs)
2025-09-07T09:09:57.1240840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1240914Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1241153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1241215Z     outputs = block(
2025-09-07T09:09:57.1241426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1241500Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1241739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1241802Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1242049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1242138Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1242424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1242536Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1242753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1242857Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1242861Z 
2025-09-07T09:09:57.1242952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1243144Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1243202Z     return mod(**inputs)
2025-09-07T09:09:57.1243453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1243524Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1243801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1243867Z     outputs = block(
2025-09-07T09:09:57.1244082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1244157Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1244390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1244450Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1244727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1244804Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1245045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1245107Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1245356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1245539Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1245747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1245856Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1245859Z 
2025-09-07T09:09:57.1245932Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1246007Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1246074Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1246142Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1246239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1246422Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1246491Z     return mod(**inputs)
2025-09-07T09:09:57.1246739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1246813Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1247059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1247114Z     outputs = block(
2025-09-07T09:09:57.1247337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1247403Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1247642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1247703Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1247944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1248087Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1248320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1248384Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1248621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1248705Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1248998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1249112Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1249115Z 
2025-09-07T09:09:57.1249207Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1249439Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1249500Z     return mod(**inputs)
2025-09-07T09:09:57.1249754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1249824Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1250071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1250125Z     outputs = block(
2025-09-07T09:09:57.1250374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1250442Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1250679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1250746Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1250990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1251069Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1251304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1251364Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1251611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1251693Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1251986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1252083Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1252086Z 
2025-09-07T09:09:57.1252179Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1252364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1252423Z     return mod(**inputs)
2025-09-07T09:09:57.1252670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1252741Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1252985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1253037Z     outputs = block(
2025-09-07T09:09:57.1253251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1253327Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1253567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1253632Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1253910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1254015Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1254256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1254317Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1254575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1254649Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1254867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1254971Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1254974Z 
2025-09-07T09:09:57.1255064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1255290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1255350Z     return mod(**inputs)
2025-09-07T09:09:57.1255602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1255675Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1255916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1255973Z     outputs = block(
2025-09-07T09:09:57.1256217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1256287Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1256521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1256579Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1256828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1256918Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1257158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1257225Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1257440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1257538Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1257541Z 
2025-09-07T09:09:57.1257701Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1257889Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1257945Z     return mod(**inputs)
2025-09-07T09:09:57.1258203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1258276Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1258518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1258578Z     outputs = block(
2025-09-07T09:09:57.1258790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1258861Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1259100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1259167Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1259410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1259539Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1259822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1259893Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1260104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1260272Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1260276Z 
2025-09-07T09:09:57.1260368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1260561Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1260619Z     return mod(**inputs)
2025-09-07T09:09:57.1260868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1260970Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1261222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1261277Z     outputs = block(
2025-09-07T09:09:57.1261488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1261556Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1261795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1261858Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1262128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1262221Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1262464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1262541Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1262753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1262853Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1262856Z 
2025-09-07T09:09:57.1262943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1263127Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1263183Z     return mod(**inputs)
2025-09-07T09:09:57.1263431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1263502Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1263747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1263803Z     outputs = block(
2025-09-07T09:09:57.1264018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1264085Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1264319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1264385Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1264625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1264724Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1264728Z 
2025-09-07T09:09:57.1264820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1265004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1265066Z     return mod(**inputs)
2025-09-07T09:09:57.1265346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1265450Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1265697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1265753Z     outputs = block(
2025-09-07T09:09:57.1265967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1266034Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1266272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1266331Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1266570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1266678Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1266918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1266978Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1267218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1267395Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1267643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1267746Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1267749Z 
2025-09-07T09:09:57.1267822Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1267891Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1267963Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1268032Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1268124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1268309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1268365Z     return mod(**inputs)
2025-09-07T09:09:57.1268613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1268683Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1268944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1269008Z     outputs = block(
2025-09-07T09:09:57.1269220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1269294Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1269536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1269600Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1269847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1269920Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1270158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1270219Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1270467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1270555Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1270846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1271003Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1271057Z 
2025-09-07T09:09:57.1271145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1271332Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1271386Z     return mod(**inputs)
2025-09-07T09:09:57.1271630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1271705Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1271946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1272003Z     outputs = block(
2025-09-07T09:09:57.1272213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1272281Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1272553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1272615Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1272860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1272933Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1273165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1273227Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1273499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1273583Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1273873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1273971Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1273976Z 
2025-09-07T09:09:57.1274068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1282614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1282725Z     return mod(**inputs)
2025-09-07T09:09:57.1283044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1283141Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1283421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1283485Z     outputs = block(
2025-09-07T09:09:57.1283732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1283817Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1284080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1284151Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1284422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1284517Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1284780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1284851Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1285120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1285202Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1285424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1285619Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1285677Z 
2025-09-07T09:09:57.1285788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1285996Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1286060Z     return mod(**inputs)
2025-09-07T09:09:57.1286321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1286404Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1286659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1286716Z     outputs = block(
2025-09-07T09:09:57.1286944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1287061Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1287383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1287452Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1287709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1287814Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1288062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1288179Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1288401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1288516Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1288520Z 
2025-09-07T09:09:57.1288627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1288832Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1288898Z     return mod(**inputs)
2025-09-07T09:09:57.1289150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1289230Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1289480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1289538Z     outputs = block(
2025-09-07T09:09:57.1289766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1289840Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1290088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1290157Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1290407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1290509Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1290760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1290838Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1291047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1291230Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1291235Z 
2025-09-07T09:09:57.1291336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1291537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1291644Z     return mod(**inputs)
2025-09-07T09:09:57.1291936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1292017Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1292265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1292322Z     outputs = block(
2025-09-07T09:09:57.1292547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1292624Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1292871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1292936Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1293215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1293320Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1293565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1293649Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1293864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1293981Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1293987Z 
2025-09-07T09:09:57.1294127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1294325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1294387Z     return mod(**inputs)
2025-09-07T09:09:57.1294641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1294723Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1294967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1295025Z     outputs = block(
2025-09-07T09:09:57.1295248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1295318Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1295561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1295624Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1295869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1295957Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1296197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1296264Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1296504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1296687Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1296910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1297019Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1297022Z 
2025-09-07T09:09:57.1297103Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1297177Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1297247Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1297315Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1297450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1297750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1297812Z     return mod(**inputs)
2025-09-07T09:09:57.1298099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1298175Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1298473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1319188Z     outputs = block(
2025-09-07T09:09:57.1319444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1319523Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1319771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1319919Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1320178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1320264Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1320504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1320563Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1320806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1320946Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1321246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1321377Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1321383Z 
2025-09-07T09:09:57.1321486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1321685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1321744Z     return mod(**inputs)
2025-09-07T09:09:57.1321997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1322075Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1322320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1322378Z     outputs = block(
2025-09-07T09:09:57.1322596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1322667Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1322909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1322972Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1323215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1323291Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1323528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1323585Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1323861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1323953Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1324242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1324346Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1324404Z 
2025-09-07T09:09:57.1324541Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1324733Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1324794Z     return mod(**inputs)
2025-09-07T09:09:57.1325043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1325118Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1325360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1325416Z     outputs = block(
2025-09-07T09:09:57.1325630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1325696Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1325993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1326061Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1326311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1326389Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1326626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1326688Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1326959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1327035Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1327249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1327363Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1327371Z 
2025-09-07T09:09:57.1327468Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1327658Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1327719Z     return mod(**inputs)
2025-09-07T09:09:57.1327964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1328038Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1328278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1328333Z     outputs = block(
2025-09-07T09:09:57.1328550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1328620Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1328862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1328922Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1329158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1329258Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1329498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1329569Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1329779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1329882Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1329887Z 
2025-09-07T09:09:57.1329976Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1330202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1330616Z     return mod(**inputs)
2025-09-07T09:09:57.1330862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1330934Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1331173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1331229Z     outputs = block(
2025-09-07T09:09:57.1331445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1331510Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1331746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1331804Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1332081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1332174Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1332413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1332485Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1332691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1332899Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1332902Z 
2025-09-07T09:09:57.1332992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1333176Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1333234Z     return mod(**inputs)
2025-09-07T09:09:57.1333481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1333555Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1333794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1333846Z     outputs = block(
2025-09-07T09:09:57.1334059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1334124Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1334362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1334421Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1334663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1334761Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1335003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1335086Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1335298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1335409Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1335412Z 
2025-09-07T09:09:57.1335506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1335697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1335756Z     return mod(**inputs)
2025-09-07T09:09:57.1336001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1336079Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1336367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1336502Z     outputs = block(
2025-09-07T09:09:57.1336719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1336788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1337031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1337092Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1337339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1337435Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1337439Z 
2025-09-07T09:09:57.1337531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1337815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1337875Z     return mod(**inputs)
2025-09-07T09:09:57.1338132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1338205Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1338448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1338505Z     outputs = block(
2025-09-07T09:09:57.1338751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1338824Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1339062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1339125Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1339372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1339453Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1339691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1339750Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1339991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1340175Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1340386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1340500Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1340504Z 
2025-09-07T09:09:57.1340582Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1340653Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1340724Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1340791Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1340886Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1341070Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1341131Z     return mod(**inputs)
2025-09-07T09:09:57.1341376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1341453Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1341693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1341746Z     outputs = block(
2025-09-07T09:09:57.1341961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1342090Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1342325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1342383Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1342620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1342696Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1342930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1342990Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1343229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1343314Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1343644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1343765Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1343767Z 
2025-09-07T09:09:57.1343857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1344040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1344096Z     return mod(**inputs)
2025-09-07T09:09:57.1344372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1344445Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1344687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1344741Z     outputs = block(
2025-09-07T09:09:57.1344957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1345025Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1345258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1345319Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1345558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1345636Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1345871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1345933Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1346175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1346259Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1346552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1346655Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1346658Z 
2025-09-07T09:09:57.1346753Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1346936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1346992Z     return mod(**inputs)
2025-09-07T09:09:57.1347242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1347313Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1347556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1347609Z     outputs = block(
2025-09-07T09:09:57.1347863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1347983Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1348218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1348280Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1348519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1348593Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1348830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1348890Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1349134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1349240Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1349458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1349564Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1349567Z 
2025-09-07T09:09:57.1349655Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1349846Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1349903Z     return mod(**inputs)
2025-09-07T09:09:57.1350183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1350256Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1350500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1350556Z     outputs = block(
2025-09-07T09:09:57.1350768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1350838Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1351071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1351131Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1351370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1351462Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1351708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1351779Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1352097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1352204Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1352209Z 
2025-09-07T09:09:57.1352299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1352486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1352544Z     return mod(**inputs)
2025-09-07T09:09:57.1352793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1352861Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1353103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1353162Z     outputs = block(
2025-09-07T09:09:57.1353375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1353444Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1353726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1353824Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1354070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1354163Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1354404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1354474Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1354681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1354852Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1354857Z 
2025-09-07T09:09:57.1354980Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1355174Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1355229Z     return mod(**inputs)
2025-09-07T09:09:57.1355478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1355549Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1355792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1355844Z     outputs = block(
2025-09-07T09:09:57.1356087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1356164Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1356401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1356471Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1356714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1356802Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1357042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1357117Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1357331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1357433Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1357437Z 
2025-09-07T09:09:57.1357527Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1357707Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1357768Z     return mod(**inputs)
2025-09-07T09:09:57.1358013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1358082Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1358324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1358376Z     outputs = block(
2025-09-07T09:09:57.1358589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1358660Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1358894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1358959Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1359198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1359313Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1359585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1359647Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1359892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1360071Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1360287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1360391Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1360394Z 
2025-09-07T09:09:57.1360467Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1360541Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1360643Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1360714Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1360801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1360983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1361041Z     return mod(**inputs)
2025-09-07T09:09:57.1361288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1361360Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1361636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1361692Z     outputs = block(
2025-09-07T09:09:57.1361911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1361977Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1362217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1362278Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1362517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1362594Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1362825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1362886Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1363125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1363213Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1363502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1363619Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1363622Z 
2025-09-07T09:09:57.1363715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1363900Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1363961Z     return mod(**inputs)
2025-09-07T09:09:57.1364205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1364276Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1364518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1364574Z     outputs = block(
2025-09-07T09:09:57.1364790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1364892Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1365163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1365221Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1365461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1365549Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1365793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1365863Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1366111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1366199Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1366528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1366633Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1366636Z 
2025-09-07T09:09:57.1366734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1366927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1366986Z     return mod(**inputs)
2025-09-07T09:09:57.1367239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1367347Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1367598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1367655Z     outputs = block(
2025-09-07T09:09:57.1367878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1367954Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1368190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1368254Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1368493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1368572Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1368808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1368866Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1369107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1369179Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1369397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1369510Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1369513Z 
2025-09-07T09:09:57.1369611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1369802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1369859Z     return mod(**inputs)
2025-09-07T09:09:57.1370106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1370183Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1370428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1370481Z     outputs = block(
2025-09-07T09:09:57.1370693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1370816Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1371082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1371147Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1371387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1371482Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1371728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1371798Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1372012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1372112Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1372117Z 
2025-09-07T09:09:57.1372241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1372434Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1372492Z     return mod(**inputs)
2025-09-07T09:09:57.1372740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1372809Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1373058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1373148Z     outputs = block(
2025-09-07T09:09:57.1373374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1373444Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1373681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1373747Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1373986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1374074Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1374315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1374384Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1374598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1374773Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1374776Z 
2025-09-07T09:09:57.1374875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1375061Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1375119Z     return mod(**inputs)
2025-09-07T09:09:57.1375371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1375441Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1375685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1375738Z     outputs = block(
2025-09-07T09:09:57.1375953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1376025Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1376258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1376327Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1376606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1376732Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1376974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1377050Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1377268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1377371Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1377376Z 
2025-09-07T09:09:57.1377470Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1377686Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1377744Z     return mod(**inputs)
2025-09-07T09:09:57.1378030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1378104Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1378349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1378403Z     outputs = block(
2025-09-07T09:09:57.1378614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1378685Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1378950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1379016Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1379259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1379359Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1379365Z 
2025-09-07T09:09:57.1379455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1379639Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1379699Z     return mod(**inputs)
2025-09-07T09:09:57.1379942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1380014Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1380257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1380311Z     outputs = block(
2025-09-07T09:09:57.1380528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1380593Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1380836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1380899Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1381137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1381219Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1381452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1381517Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1381758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1381945Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1382153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1382293Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1382331Z 
2025-09-07T09:09:57.1382410Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1382479Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1382550Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1382616Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1382701Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1382890Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1382947Z     return mod(**inputs)
2025-09-07T09:09:57.1383199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1383270Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1383510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1383602Z     outputs = block(
2025-09-07T09:09:57.1383817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1383889Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1384124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1384189Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1384428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1384534Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1384776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1384834Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1385078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1385170Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1385465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1385585Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1385588Z 
2025-09-07T09:09:57.1385677Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1385864Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1385923Z     return mod(**inputs)
2025-09-07T09:09:57.1386168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1386236Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1386490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1386554Z     outputs = block(
2025-09-07T09:09:57.1386782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1386855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1387100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1387160Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1387407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1387482Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1387721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1387779Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1388063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1388186Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1388482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1388586Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1388589Z 
2025-09-07T09:09:57.1388682Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1388886Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1388947Z     return mod(**inputs)
2025-09-07T09:09:57.1389208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1389284Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1389557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1389617Z     outputs = block(
2025-09-07T09:09:57.1389832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1389902Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1390142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1390200Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1390487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1390561Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1390800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1390862Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1391104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1391181Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1391390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1391504Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1391507Z 
2025-09-07T09:09:57.1391599Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1391788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1391851Z     return mod(**inputs)
2025-09-07T09:09:57.1392096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1392169Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1392411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1392468Z     outputs = block(
2025-09-07T09:09:57.1392683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1392752Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1392990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1393046Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1393287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1393384Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1393624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1393737Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1393952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1394089Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1394092Z 
2025-09-07T09:09:57.1394180Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1394364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1394427Z     return mod(**inputs)
2025-09-07T09:09:57.1394685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1394760Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1394999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1395053Z     outputs = block(
2025-09-07T09:09:57.1395321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1395397Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1395646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1395706Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1395951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1396040Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1396313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1396388Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1396596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1396779Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1396783Z 
2025-09-07T09:09:57.1396873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1397057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1397118Z     return mod(**inputs)
2025-09-07T09:09:57.1397362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1397440Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1397679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1397738Z     outputs = block(
2025-09-07T09:09:57.1397952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1398023Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1398265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1398327Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1398573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1398659Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1398905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1398985Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1399194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1399305Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1399308Z 
2025-09-07T09:09:57.1399437Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1399624Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1399718Z     return mod(**inputs)
2025-09-07T09:09:57.1399967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1400046Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1400286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1400346Z     outputs = block(
2025-09-07T09:09:57.1400560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1400630Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1400873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1400966Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1401220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1401299Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1401536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1401603Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1401844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1402064Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1402283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1402398Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1402403Z 
2025-09-07T09:09:57.1402485Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1402556Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1402630Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1402698Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1402801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1402996Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1403184Z     return mod(**inputs)
2025-09-07T09:09:57.1403448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1403523Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1403775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1403830Z     outputs = block(
2025-09-07T09:09:57.1404052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1404130Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1404373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1404441Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1404682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1404762Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1405004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1405065Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1405312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1405460Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1405763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1405939Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1405942Z 
2025-09-07T09:09:57.1406035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1406228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1406286Z     return mod(**inputs)
2025-09-07T09:09:57.1406542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1406615Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1406856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1406916Z     outputs = block(
2025-09-07T09:09:57.1407178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1407260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1407495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1407555Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1407802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1407877Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1408174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1408233Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1408481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1408569Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1408861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1408966Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1408969Z 
2025-09-07T09:09:57.1409064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1409256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1409313Z     return mod(**inputs)
2025-09-07T09:09:57.1409562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1409642Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1409886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1409950Z     outputs = block(
2025-09-07T09:09:57.1410162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1410238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1410474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1410535Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1410780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1410857Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1411098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1411157Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1411400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1411514Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1411761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1411874Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1411876Z 
2025-09-07T09:09:57.1411966Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1412150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1412214Z     return mod(**inputs)
2025-09-07T09:09:57.1412461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1412541Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1412782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1412902Z     outputs = block(
2025-09-07T09:09:57.1413123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1413194Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1413435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1413496Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1413741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1413869Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1414112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1414190Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1414400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1414514Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1414517Z 
2025-09-07T09:09:57.1414608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1414797Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1414855Z     return mod(**inputs)
2025-09-07T09:09:57.1415101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1415177Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1415417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1415476Z     outputs = block(
2025-09-07T09:09:57.1415689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1415759Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1415999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1416061Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1416306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1416397Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1416635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1416714Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1416930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1417117Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1417158Z 
2025-09-07T09:09:57.1417257Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1417493Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1417551Z     return mod(**inputs)
2025-09-07T09:09:57.1417878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1417965Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1418214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1418278Z     outputs = block(
2025-09-07T09:09:57.1418494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1418564Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1418852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1418918Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1419168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1419262Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1419505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1419592Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1419836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1419952Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1419955Z 
2025-09-07T09:09:57.1420050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1420245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1420303Z     return mod(**inputs)
2025-09-07T09:09:57.1420547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1420629Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1420871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1420932Z     outputs = block(
2025-09-07T09:09:57.1421145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1421217Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1421459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1421520Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1421765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1421867Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1421870Z 
2025-09-07T09:09:57.1421965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1422152Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1422207Z     return mod(**inputs)
2025-09-07T09:09:57.1422457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1422530Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1422778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1422833Z     outputs = block(
2025-09-07T09:09:57.1423046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1423157Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1423428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1423495Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1423734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1423813Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1424055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1424113Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1424359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1424543Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1424796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1424902Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1424905Z 
2025-09-07T09:09:57.1424977Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1425048Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1425114Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1425186Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1425276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1425504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1425568Z     return mod(**inputs)
2025-09-07T09:09:57.1425834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1425910Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1426151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1426207Z     outputs = block(
2025-09-07T09:09:57.1426422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1426489Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1426733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1426793Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1427035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1427113Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1427354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1427418Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1427657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1427745Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1428043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1428160Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1428163Z 
2025-09-07T09:09:57.1428254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1428437Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1428495Z     return mod(**inputs)
2025-09-07T09:09:57.1428737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1428868Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1429144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1429198Z     outputs = block(
2025-09-07T09:09:57.1429416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1429483Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1429719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1429782Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1430023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1430104Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1430368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1430445Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1430684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1430771Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1431068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1431169Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1431172Z 
2025-09-07T09:09:57.1431299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1431486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1431550Z     return mod(**inputs)
2025-09-07T09:09:57.1431799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1431878Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1432126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1432179Z     outputs = block(
2025-09-07T09:09:57.1432399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1432466Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1432700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1432770Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1433011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1433095Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1433332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1433397Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1433648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1433719Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1433937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1434041Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1434044Z 
2025-09-07T09:09:57.1434141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1434323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1434380Z     return mod(**inputs)
2025-09-07T09:09:57.1434629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1434739Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1435034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1435087Z     outputs = block(
2025-09-07T09:09:57.1435300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1435372Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1435607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1435674Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1435915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1436009Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1436288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1436363Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1436579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1436682Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1436685Z 
2025-09-07T09:09:57.1436782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1436967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1437058Z     return mod(**inputs)
2025-09-07T09:09:57.1437315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1437386Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1437635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1437695Z     outputs = block(
2025-09-07T09:09:57.1437914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1437986Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1438219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1438286Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1438526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1438616Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1438860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1438929Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1439144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1439316Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1439319Z 
2025-09-07T09:09:57.1439417Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1439603Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1439661Z     return mod(**inputs)
2025-09-07T09:09:57.1439913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1439983Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1440233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1440290Z     outputs = block(
2025-09-07T09:09:57.1440546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1440654Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1440892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1440961Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1441203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1441304Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1441549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1441629Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1441846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1441985Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1441990Z 
2025-09-07T09:09:57.1442091Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1442281Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1442341Z     return mod(**inputs)
2025-09-07T09:09:57.1442595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1442667Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1442949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1443006Z     outputs = block(
2025-09-07T09:09:57.1443224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1443294Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1443532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1443599Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1443839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1443919Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1444154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1444215Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1444465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1444646Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1444860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1444964Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1444967Z 
2025-09-07T09:09:57.1445046Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1445118Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1445185Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1445261Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1445353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1445537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1445602Z     return mod(**inputs)
2025-09-07T09:09:57.1445847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1445926Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1446169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1446293Z     outputs = block(
2025-09-07T09:09:57.1446510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1446579Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1446822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1446887Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1447133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1447208Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1447449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1447515Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1447788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1447884Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1448178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1448294Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1448303Z 
2025-09-07T09:09:57.1448394Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1448608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1448672Z     return mod(**inputs)
2025-09-07T09:09:57.1448917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1448996Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1449239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1449295Z     outputs = block(
2025-09-07T09:09:57.1449514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1449581Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1449822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1449882Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1450124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1450204Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1450439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1450508Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1450748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1450836Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1451125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1451226Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1451229Z 
2025-09-07T09:09:57.1451324Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1451507Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1451571Z     return mod(**inputs)
2025-09-07T09:09:57.1451816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1451887Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1452174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1452260Z     outputs = block(
2025-09-07T09:09:57.1452481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1452547Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1452784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1452848Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1453091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1453171Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1453404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1453503Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1453750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1453827Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1454044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1454151Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1454154Z 
2025-09-07T09:09:57.1454250Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1454466Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1454526Z     return mod(**inputs)
2025-09-07T09:09:57.1454777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1454847Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1455094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1455149Z     outputs = block(
2025-09-07T09:09:57.1455364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1455437Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1455671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1455736Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1455976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1456071Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1456314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1456389Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1456604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1456704Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1456708Z 
2025-09-07T09:09:57.1456801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1456984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1457042Z     return mod(**inputs)
2025-09-07T09:09:57.1457293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1457363Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1457665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1457776Z     outputs = block(
2025-09-07T09:09:57.1457998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1458102Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1458339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1458405Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1458643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1458739Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1458981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1459051Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1459262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1459473Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1459478Z 
2025-09-07T09:09:57.1459577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1459762Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1459824Z     return mod(**inputs)
2025-09-07T09:09:57.1460072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1460173Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1460420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1460474Z     outputs = block(
2025-09-07T09:09:57.1460696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1460765Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1460997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1461064Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1461304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1461395Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1461636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1461713Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1461924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1462026Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1462031Z 
2025-09-07T09:09:57.1462128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1462314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1462375Z     return mod(**inputs)
2025-09-07T09:09:57.1462618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1462690Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1462936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1462992Z     outputs = block(
2025-09-07T09:09:57.1463208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1463275Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1463511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1463613Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1463884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1463985Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1463988Z 
2025-09-07T09:09:57.1464077Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1464265Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1464323Z     return mod(**inputs)
2025-09-07T09:09:57.1464569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1464645Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1464885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1464980Z     outputs = block(
2025-09-07T09:09:57.1465195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1465266Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1465506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1465568Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1465812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1465922Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1466165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1466233Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1466475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1466663Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1466875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1466985Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1466989Z 
2025-09-07T09:09:57.1467061Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1467129Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1467206Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1467277Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1467373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1467559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1467616Z     return mod(**inputs)
2025-09-07T09:09:57.1467879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1467954Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1468201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1468258Z     outputs = block(
2025-09-07T09:09:57.1468469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1468541Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1468778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1468843Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1469081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1469155Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1469433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1469525Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1469775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1469861Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1470162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1470280Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1470283Z 
2025-09-07T09:09:57.1470373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1470562Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1470618Z     return mod(**inputs)
2025-09-07T09:09:57.1470904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1470978Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1471218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1471277Z     outputs = block(
2025-09-07T09:09:57.1471487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1471558Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1471822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1471882Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1472130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1472209Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1472450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1472510Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1472755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1472841Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1473134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1473239Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1473242Z 
2025-09-07T09:09:57.1473331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1473521Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1473581Z     return mod(**inputs)
2025-09-07T09:09:57.1473837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1473917Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1474156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1474219Z     outputs = block(
2025-09-07T09:09:57.1474431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1474507Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1474741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1474802Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1475050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1475162Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1475443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1475504Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1475745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1475824Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1476033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1476144Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1476147Z 
2025-09-07T09:09:57.1476242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1476435Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1476529Z     return mod(**inputs)
2025-09-07T09:09:57.1476778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1476862Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1477101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1477160Z     outputs = block(
2025-09-07T09:09:57.1477373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1477491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1477734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1477795Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1478040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1478138Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1478378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1478453Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1478664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1478774Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1478777Z 
2025-09-07T09:09:57.1478871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1479067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1479125Z     return mod(**inputs)
2025-09-07T09:09:57.1479372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1479453Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1479694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1479754Z     outputs = block(
2025-09-07T09:09:57.1479965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1480031Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1480271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1480332Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1480576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1480662Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1480905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1481042Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1481247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1481424Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1481427Z 
2025-09-07T09:09:57.1481517Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1481706Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1481766Z     return mod(**inputs)
2025-09-07T09:09:57.1482011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1482092Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1482372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1482435Z     outputs = block(
2025-09-07T09:09:57.1482648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1482712Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1482954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1483014Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1483291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1483378Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1483625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1483703Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1483916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1484027Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1484030Z 
2025-09-07T09:09:57.1484117Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1484308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1484366Z     return mod(**inputs)
2025-09-07T09:09:57.1484612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1484690Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1484930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1484992Z     outputs = block(
2025-09-07T09:09:57.1485204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1485274Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1485515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1485576Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1485828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1485906Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1486146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1486206Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1486444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1486629Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1486906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1487012Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1487015Z 
2025-09-07T09:09:57.1487086Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1487155Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1487226Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1487293Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1487388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1487572Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1487627Z     return mod(**inputs)
2025-09-07T09:09:57.1487882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1487987Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1488239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1488294Z     outputs = block(
2025-09-07T09:09:57.1488510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1488582Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1488817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1488917Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1489160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1489243Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1489478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1489542Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1489785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1489869Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1490165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1490278Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1490281Z 
2025-09-07T09:09:57.1490373Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1490564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1490623Z     return mod(**inputs)
2025-09-07T09:09:57.1490873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1490946Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1491188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1491245Z     outputs = block(
2025-09-07T09:09:57.1491457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1491530Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1491765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1491833Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1492071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1492144Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1492420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1492514Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1492764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1492845Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1493133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1493236Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1493241Z 
2025-09-07T09:09:57.1493334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1493524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1493581Z     return mod(**inputs)
2025-09-07T09:09:57.1493863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1493934Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1494177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1494242Z     outputs = block(
2025-09-07T09:09:57.1494459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1494534Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1494810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1494875Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1495128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1495201Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1495445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1495504Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1495750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1495822Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1496030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1496142Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1496146Z 
2025-09-07T09:09:57.1496234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1496424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1496480Z     return mod(**inputs)
2025-09-07T09:09:57.1496726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1496806Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1497042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1497099Z     outputs = block(
2025-09-07T09:09:57.1497310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1497375Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1497683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1497745Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1497994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1498085Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1498371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1498474Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1498681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1498788Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1498791Z 
2025-09-07T09:09:57.1498878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1499066Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1499124Z     return mod(**inputs)
2025-09-07T09:09:57.1499365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1499442Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1499730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1499796Z     outputs = block(
2025-09-07T09:09:57.1500010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1500076Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1500315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1500373Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1500668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1500759Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1501006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1501080Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1501288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1501463Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1501465Z 
2025-09-07T09:09:57.1501555Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1501739Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1501794Z     return mod(**inputs)
2025-09-07T09:09:57.1502040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1502118Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1502358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1502422Z     outputs = block(
2025-09-07T09:09:57.1502638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1502712Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1502948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1503009Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1503419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1503517Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1503766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1503842Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1504054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1504225Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1504274Z 
2025-09-07T09:09:57.1504367Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1504561Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1504619Z     return mod(**inputs)
2025-09-07T09:09:57.1504877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1504951Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1505192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1505253Z     outputs = block(
2025-09-07T09:09:57.1505465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1505583Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1505826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1505890Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1506145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1506242Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1506245Z 
2025-09-07T09:09:57.1506346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1506575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1506635Z     return mod(**inputs)
2025-09-07T09:09:57.1506886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1506960Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1507210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1507266Z     outputs = block(
2025-09-07T09:09:57.1507489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1507560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1507800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1507870Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1508115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1508200Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1508433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1508497Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1508746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1508925Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1509151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1509258Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1509261Z 
2025-09-07T09:09:57.1509342Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1509409Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1509476Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1509546Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1509637Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1509831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1509961Z     return mod(**inputs)
2025-09-07T09:09:57.1510206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1510283Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1510526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1510583Z     outputs = block(
2025-09-07T09:09:57.1510801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1510867Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1511107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1511167Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1511447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1511525Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1511760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1511822Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1512080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1512174Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1512501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1512631Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1512633Z 
2025-09-07T09:09:57.1512729Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1512923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1512987Z     return mod(**inputs)
2025-09-07T09:09:57.1513238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1513317Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1513564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1513618Z     outputs = block(
2025-09-07T09:09:57.1513839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1513909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1514150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1514212Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1514458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1514543Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1514778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1514842Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1515081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1515176Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1515466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1515567Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1515570Z 
2025-09-07T09:09:57.1515668Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1515893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1515989Z     return mod(**inputs)
2025-09-07T09:09:57.1516239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1516316Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1516562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1516619Z     outputs = block(
2025-09-07T09:09:57.1516843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1516912Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1517150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1517223Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1517495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1517580Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1517818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1517883Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1518123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1518252Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1518475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1518582Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1518586Z 
2025-09-07T09:09:57.1518683Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1518873Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1518934Z     return mod(**inputs)
2025-09-07T09:09:57.1519186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1519259Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1519508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1519563Z     outputs = block(
2025-09-07T09:09:57.1519782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1519851Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1520090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1520160Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1520400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1520505Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1520745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1520816Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1521033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1521135Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1521138Z 
2025-09-07T09:09:57.1521236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1521419Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1521514Z     return mod(**inputs)
2025-09-07T09:09:57.1521768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1521887Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1522134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1522190Z     outputs = block(
2025-09-07T09:09:57.1522410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1522476Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1522711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1522776Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1523015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1523146Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1523392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1523465Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1523678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1523851Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1523854Z 
2025-09-07T09:09:57.1523988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1524178Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1524239Z     return mod(**inputs)
2025-09-07T09:09:57.1524484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1524560Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1524807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1524862Z     outputs = block(
2025-09-07T09:09:57.1525081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1525150Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1525390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1525455Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1525697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1525790Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1526031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1526110Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1526326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1530094Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1530105Z 
2025-09-07T09:09:57.1530244Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1530460Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1530528Z     return mod(**inputs)
2025-09-07T09:09:57.1530816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1530896Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1531165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1531321Z     outputs = block(
2025-09-07T09:09:57.1531558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1531634Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1531882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1531948Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1532200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1532291Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1532539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1532602Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1532888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1533079Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1533300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1533418Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1533422Z 
2025-09-07T09:09:57.1533500Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1533567Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1533665Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1533738Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1533836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1534038Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1534101Z     return mod(**inputs)
2025-09-07T09:09:57.1534359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1534442Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1534686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1534743Z     outputs = block(
2025-09-07T09:09:57.1534960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1535034Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1535286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1535349Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1535599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1535685Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1535931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1535997Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1536240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1536332Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1536628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1536753Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1536757Z 
2025-09-07T09:09:57.1536852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1537048Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1537150Z     return mod(**inputs)
2025-09-07T09:09:57.1537434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1537514Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1537825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1537882Z     outputs = block(
2025-09-07T09:09:57.1538103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1538176Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1538420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1538480Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1538756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1538836Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1539072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1539134Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1539373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1539459Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1539782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1539885Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1539888Z 
2025-09-07T09:09:57.1539986Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1540177Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1540240Z     return mod(**inputs)
2025-09-07T09:09:57.1540490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1540564Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1540810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1540876Z     outputs = block(
2025-09-07T09:09:57.1541094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1541164Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1541405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1541465Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1541704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1541785Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1542017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1542082Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1542320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1542394Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1542612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1542721Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1542725Z 
2025-09-07T09:09:57.1542822Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1543011Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1543109Z     return mod(**inputs)
2025-09-07T09:09:57.1543392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1543466Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1543710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1543764Z     outputs = block(
2025-09-07T09:09:57.1543979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1544047Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1544282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1544343Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1544620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1544724Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1544962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1545029Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1545245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1545346Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1545349Z 
2025-09-07T09:09:57.1545474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1545661Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1545719Z     return mod(**inputs)
2025-09-07T09:09:57.1545971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1546044Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1546284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1546339Z     outputs = block(
2025-09-07T09:09:57.1546553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1546619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1546856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1546920Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1547160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1547248Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1547489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1547558Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1547765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1547939Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1547944Z 
2025-09-07T09:09:57.1548034Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1548218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1548275Z     return mod(**inputs)
2025-09-07T09:09:57.1548521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1548591Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1548883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1548968Z     outputs = block(
2025-09-07T09:09:57.1549183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1549250Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1549483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1549544Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1549783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1549874Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1550113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1550224Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1550558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1550663Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1550666Z 
2025-09-07T09:09:57.1550760Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1550943Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1551001Z     return mod(**inputs)
2025-09-07T09:09:57.1551281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1551353Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1551594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1551648Z     outputs = block(
2025-09-07T09:09:57.1551866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1551936Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1552169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1552233Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1552469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1552568Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1552573Z 
2025-09-07T09:09:57.1552659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1552843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1552898Z     return mod(**inputs)
2025-09-07T09:09:57.1553141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1553216Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1553455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1553512Z     outputs = block(
2025-09-07T09:09:57.1553723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1553788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1554027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1554085Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1554330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1554405Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1554680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1554778Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1555017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1555203Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1555416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1555526Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1555529Z 
2025-09-07T09:09:57.1555601Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1555668Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1555740Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1555804Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1556159Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1556352Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1556413Z     return mod(**inputs)
2025-09-07T09:09:57.1556666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1556739Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1556984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1557074Z     outputs = block(
2025-09-07T09:09:57.1557295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1557373Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1557615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1557683Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1557925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1558001Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1558240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1558301Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1558545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1558630Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1558925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1559044Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1559051Z 
2025-09-07T09:09:57.1559144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1559332Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1559388Z     return mod(**inputs)
2025-09-07T09:09:57.1559639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1559711Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1559953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1560014Z     outputs = block(
2025-09-07T09:09:57.1560227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1560298Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1560532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1560661Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1560913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1560985Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1561224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1561284Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1561528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1561608Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1561892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1562031Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1562036Z 
2025-09-07T09:09:57.1562126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1562313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1562366Z     return mod(**inputs)
2025-09-07T09:09:57.1562610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1562682Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1562950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1563005Z     outputs = block(
2025-09-07T09:09:57.1563216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1563285Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1563520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1563581Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1563820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1563893Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1564128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1564186Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1564428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1564502Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1564711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1564820Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1564824Z 
2025-09-07T09:09:57.1564912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1565095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1565150Z     return mod(**inputs)
2025-09-07T09:09:57.1565394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1565465Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1565703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1565757Z     outputs = block(
2025-09-07T09:09:57.1565969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1566033Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1566305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1566400Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1566643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1566736Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1566976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1567047Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1567257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1567364Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1567368Z 
2025-09-07T09:09:57.1567456Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1567676Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1567733Z     return mod(**inputs)
2025-09-07T09:09:57.1567975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1568051Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1568289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1568343Z     outputs = block(
2025-09-07T09:09:57.1568599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1568665Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1568903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1568964Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1569206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1569297Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1569534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1569608Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1569812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1569986Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1569989Z 
2025-09-07T09:09:57.1570076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1570261Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1570319Z     return mod(**inputs)
2025-09-07T09:09:57.1570565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1570639Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1570881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1570939Z     outputs = block(
2025-09-07T09:09:57.1571152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1571219Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1571470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1571532Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1571777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1571904Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1572191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1572269Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1572478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1572590Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1572594Z 
2025-09-07T09:09:57.1572680Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1572877Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1572937Z     return mod(**inputs)
2025-09-07T09:09:57.1573181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1573296Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1573542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1573606Z     outputs = block(
2025-09-07T09:09:57.1573818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1573888Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1574132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1574195Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1574477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1574554Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1574804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1574866Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1575110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1575300Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1575537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1575655Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1575658Z 
2025-09-07T09:09:57.1575732Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1575803Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1575879Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1575946Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1576046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1576243Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1576306Z     return mod(**inputs)
2025-09-07T09:09:57.1576569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1576645Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1576898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1576955Z     outputs = block(
2025-09-07T09:09:57.1577178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1577250Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1577488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1577556Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1577875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1577992Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1578230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1578293Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1578543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1578628Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1578927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1579047Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1579050Z 
2025-09-07T09:09:57.1579152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1579375Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1579436Z     return mod(**inputs)
2025-09-07T09:09:57.1579695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1579770Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1580027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1580083Z     outputs = block(
2025-09-07T09:09:57.1580331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1580404Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1580638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1580706Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1580946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1581021Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1581266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1581326Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1581583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1581666Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1581959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1582059Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1582063Z 
2025-09-07T09:09:57.1582155Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1582349Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1582406Z     return mod(**inputs)
2025-09-07T09:09:57.1582657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1582727Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1582967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1583027Z     outputs = block(
2025-09-07T09:09:57.1583239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1583314Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1583549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1583648Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1583925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1584000Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1584237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1584294Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1584536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1584611Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1584821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1584929Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1584934Z 
2025-09-07T09:09:57.1585056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1585245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1585300Z     return mod(**inputs)
2025-09-07T09:09:57.1585548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1585625Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1585868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1585960Z     outputs = block(
2025-09-07T09:09:57.1586172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1586238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1586480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1586542Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1586783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1586874Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1587118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1587186Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1587394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1587501Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1587504Z 
2025-09-07T09:09:57.1587594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1587783Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1587842Z     return mod(**inputs)
2025-09-07T09:09:57.1588088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1588163Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1588401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1588457Z     outputs = block(
2025-09-07T09:09:57.1588668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1588737Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1588969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1589027Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1589268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1589420Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1589660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1589728Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1589934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1590111Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1590114Z 
2025-09-07T09:09:57.1590205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1590391Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1590446Z     return mod(**inputs)
2025-09-07T09:09:57.1590743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1590821Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1591061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1591123Z     outputs = block(
2025-09-07T09:09:57.1591333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1591403Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1591670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1591731Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1591978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1592065Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1592309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1592387Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1592597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1592707Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1592710Z 
2025-09-07T09:09:57.1592799Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1592993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1593049Z     return mod(**inputs)
2025-09-07T09:09:57.1593295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1593369Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1593612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1593674Z     outputs = block(
2025-09-07T09:09:57.1593891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1593963Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1594200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1594261Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1594503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward
2025-09-07T09:09:57.1594601Z     hidden_states = residual + feed_forward_hidden_states
2025-09-07T09:09:57.1594604Z 
2025-09-07T09:09:57.1594697Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1594883Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1594989Z     return mod(**inputs)
2025-09-07T09:09:57.1595274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1595347Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1595592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1595645Z     outputs = block(
2025-09-07T09:09:57.1595863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1595933Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1596167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1596230Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1596503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1596588Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1596824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1596883Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1597130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward
2025-09-07T09:09:57.1597309Z     query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
2025-09-07T09:09:57.1597561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1597665Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1597668Z 
2025-09-07T09:09:57.1597743Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1597812Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1597879Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1597950Z cudagraph partition due to non gpu ops
2025-09-07T09:09:57.1598039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1598230Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1598289Z     return mod(**inputs)
2025-09-07T09:09:57.1598533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1598608Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1598848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1598904Z     outputs = block(
2025-09-07T09:09:57.1599117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1599187Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1599428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1599486Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1599731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1599803Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1600036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1600101Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1600341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1600433Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1600727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:09:57.1600910Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:09:57.1600913Z 
2025-09-07T09:09:57.1601002Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1601188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1601254Z     return mod(**inputs)
2025-09-07T09:09:57.1601499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1601576Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1601816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1601869Z     outputs = block(
2025-09-07T09:09:57.1602082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1602185Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1602428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1602485Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1602725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1602802Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1603240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1603306Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1603547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward
2025-09-07T09:09:57.1603630Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:09:57.1603921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:09:57.1604022Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:09:57.1604026Z 
2025-09-07T09:09:57.1604119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1604302Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1604363Z     return mod(**inputs)
2025-09-07T09:09:57.1604606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1604678Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1604919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1604973Z     outputs = block(
2025-09-07T09:09:57.1605188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1605259Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1605494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1605552Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1605792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward
2025-09-07T09:09:57.1605870Z     attn_output, self_attn_weights = self.attn(
2025-09-07T09:09:57.1606103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1606166Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1606403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward
2025-09-07T09:09:57.1606474Z     attn_output = self.c_proj(attn_output)
2025-09-07T09:09:57.1606753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1606907Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1606910Z 
2025-09-07T09:09:57.1607015Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1607209Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1607267Z     return mod(**inputs)
2025-09-07T09:09:57.1607521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1607599Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1607846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1607900Z     outputs = block(
2025-09-07T09:09:57.1608162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1608237Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1608478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1608546Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1608788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1608885Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1609161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward
2025-09-07T09:09:57.1609230Z     hidden_states = self.c_fc(hidden_states)
2025-09-07T09:09:57.1609444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1609549Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1609554Z 
2025-09-07T09:09:57.1609651Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1609837Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1609897Z     return mod(**inputs)
2025-09-07T09:09:57.1610142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1610211Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1610454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1610508Z     outputs = block(
2025-09-07T09:09:57.1610724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1610787Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1611025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1611088Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1611327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1611418Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1611656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward
2025-09-07T09:09:57.1611728Z     hidden_states = self.act(hidden_states)
2025-09-07T09:09:57.1611940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward
2025-09-07T09:09:57.1612113Z     return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0))))
2025-09-07T09:09:57.1612117Z 
2025-09-07T09:09:57.1612209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1612431Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1612535Z     return mod(**inputs)
2025-09-07T09:09:57.1612779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward
2025-09-07T09:09:57.1612850Z     transformer_outputs = self.transformer(
2025-09-07T09:09:57.1613094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward
2025-09-07T09:09:57.1613148Z     outputs = block(
2025-09-07T09:09:57.1613362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:09:57.1613428Z     return super().__call__(*args, **kwargs)
2025-09-07T09:09:57.1613684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:09:57.1613748Z     return func(*args, **kwargs)
2025-09-07T09:09:57.1614043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward
2025-09-07T09:09:57.1614135Z     feed_forward_hidden_states = self.mlp(hidden_states)
2025-09-07T09:09:57.1614373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward
2025-09-07T09:09:57.1614451Z     hidden_states = self.c_proj(hidden_states)
2025-09-07T09:09:57.1614661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward
2025-09-07T09:09:57.1614795Z     x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2025-09-07T09:09:57.1614798Z 
2025-09-07T09:09:57.1614892Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:09:57.1615076Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:09:57.1615133Z     return mod(**inputs)
2025-09-07T09:09:57.1615381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward
2025-09-07T09:09:57.1615485Z     logits = self.lm_head(hidden_states[:, slice_indices, :])
2025-09-07T09:09:57.1615488Z 
2025-09-07T09:10:19.3896630Z 
2025-09-07T09:10:20.4623857Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:10:21.4492129Z running benchmark:   3% 1/30 [00:01<00:31,  1.07s/it]
2025-09-07T09:10:22.4338971Z running benchmark:   7% 2/30 [00:02<00:28,  1.02s/it]
2025-09-07T09:10:23.4126083Z running benchmark:  10% 3/30 [00:03<00:27,  1.01s/it]
2025-09-07T09:10:24.4789724Z running benchmark:  13% 4/30 [00:04<00:25,  1.01it/s]
2025-09-07T09:10:25.4653128Z running benchmark:  17% 5/30 [00:05<00:25,  1.02s/it]
2025-09-07T09:10:26.4516371Z running benchmark:  20% 6/30 [00:06<00:24,  1.01s/it]
2025-09-07T09:10:27.4417506Z running benchmark:  23% 7/30 [00:07<00:23,  1.00s/it]
2025-09-07T09:10:28.5116317Z running benchmark:  27% 8/30 [00:08<00:21,  1.00it/s]
2025-09-07T09:10:29.4944432Z running benchmark:  30% 9/30 [00:09<00:21,  1.02s/it]
2025-09-07T09:10:30.4750478Z running benchmark:  33% 10/30 [00:10<00:20,  1.01s/it]
2025-09-07T09:10:31.4663141Z running benchmark:  37% 11/30 [00:11<00:19,  1.00s/it]
2025-09-07T09:10:32.5221032Z running benchmark:  40% 12/30 [00:12<00:17,  1.00it/s]
2025-09-07T09:10:33.5021131Z running benchmark:  43% 13/30 [00:13<00:17,  1.02s/it]
2025-09-07T09:10:34.4863802Z running benchmark:  47% 14/30 [00:14<00:16,  1.00s/it]
2025-09-07T09:10:35.4752142Z running benchmark:  50% 15/30 [00:15<00:14,  1.00it/s]
2025-09-07T09:10:36.5080001Z running benchmark:  53% 16/30 [00:16<00:13,  1.00it/s]
2025-09-07T09:10:37.4921825Z running benchmark:  57% 17/30 [00:17<00:13,  1.01s/it]
2025-09-07T09:10:38.4769687Z running benchmark:  60% 18/30 [00:18<00:11,  1.00it/s]
2025-09-07T09:10:39.4611690Z running benchmark:  63% 19/30 [00:19<00:10,  1.00it/s]
2025-09-07T09:10:40.5178430Z running benchmark:  67% 20/30 [00:20<00:09,  1.01it/s]
2025-09-07T09:10:41.5021416Z running benchmark:  70% 21/30 [00:21<00:09,  1.01s/it]
2025-09-07T09:10:42.4879329Z running benchmark:  73% 22/30 [00:22<00:08,  1.00s/it]
2025-09-07T09:10:43.4793490Z running benchmark:  77% 23/30 [00:23<00:06,  1.00it/s]
2025-09-07T09:10:44.5170375Z running benchmark:  80% 24/30 [00:24<00:05,  1.00it/s]
2025-09-07T09:10:45.5022544Z running benchmark:  83% 25/30 [00:25<00:05,  1.01s/it]
2025-09-07T09:10:46.4850442Z running benchmark:  87% 26/30 [00:26<00:04,  1.00s/it]
2025-09-07T09:10:47.4796978Z running benchmark:  90% 27/30 [00:27<00:02,  1.00it/s]
2025-09-07T09:10:48.5171246Z running benchmark:  93% 28/30 [00:28<00:01,  1.00it/s]
2025-09-07T09:10:49.5021027Z running benchmark:  97% 29/30 [00:29<00:01,  1.01s/it]
2025-09-07T09:10:49.5021332Z running benchmark: 100% 30/30 [00:30<00:00,  1.00s/it]
2025-09-07T09:10:49.5021581Z running benchmark: 100% 30/30 [00:30<00:00,  1.00s/it]
2025-09-07T09:10:49.7357825Z 1.016x
2025-09-07T09:10:49.7362689Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:10:52.9397691Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:10:52.9398566Z   import pynvml  # type: ignore[import]
2025-09-07T09:10:54.9448669Z 
2025-09-07T09:10:58.4330723Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:10:58.4331049Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:10:58.4564439Z cpu  eval  hf_Longformer                      
2025-09-07T09:10:59.6552495Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T09:10:59.6553218Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T09:10:59.6553763Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T09:10:59.6554279Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T09:10:59.6554865Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T09:10:59.6555343Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:10:59.6555790Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T09:10:59.6556461Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:10:59.6557116Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(**inputs)
2025-09-07T09:10:59.6557884Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T09:10:59.6558681Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.longformer(
2025-09-07T09:10:59.6559430Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T09:10:59.6560178Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     encoder_outputs = self.encoder(
2025-09-07T09:10:59.6560933Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1244, in forward
2025-09-07T09:10:59.6561927Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2025-09-07T09:10:59.6562435Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:10:59.6562822Z W0907 09:10:59.654415 174416 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:11:01.6892102Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:01.6892709Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1596, in forward
2025-09-07T09:11:01.6893168Z     embedding_output = self.embeddings(
2025-09-07T09:11:01.6893625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 410, in forward
2025-09-07T09:11:01.6895397Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
2025-09-07T09:11:01.6896044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 381, in create_position_ids_from_input_ids
2025-09-07T09:11:01.6896573Z     mask = input_ids.ne(padding_idx).int()
2025-09-07T09:11:01.6896720Z 
2025-09-07T09:11:01.6896835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:01.6897308Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1592, in forward
2025-09-07T09:11:01.6898029Z     extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape)[
2025-09-07T09:11:01.6898585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_utils.py", line 1763, in get_extended_attention_mask
2025-09-07T09:11:01.6899086Z     extended_attention_mask = (1.0 - extended_attention_mask) * torch.finfo(dtype).min
2025-09-07T09:11:01.6899326Z 
2025-09-07T09:11:01.6899432Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:01.6899885Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1596, in forward
2025-09-07T09:11:01.6900313Z     embedding_output = self.embeddings(
2025-09-07T09:11:01.6900728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 410, in forward
2025-09-07T09:11:01.6901279Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
2025-09-07T09:11:01.6901884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 382, in create_position_ids_from_input_ids
2025-09-07T09:11:01.6902439Z     incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
2025-09-07T09:11:01.6902635Z 
2025-09-07T09:11:01.6902742Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:01.6903416Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1596, in forward
2025-09-07T09:11:01.6903853Z     embedding_output = self.embeddings(
2025-09-07T09:11:01.6904275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 410, in forward
2025-09-07T09:11:01.6904809Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx).to(input_ids.device)
2025-09-07T09:11:01.6905415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 382, in create_position_ids_from_input_ids
2025-09-07T09:11:01.6905955Z     incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
2025-09-07T09:11:01.6906153Z 
2025-09-07T09:11:22.1335089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1338735Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1339676Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1340136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1340530Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1341004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1341489Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1341930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1342358Z     self_outputs = self.self(
2025-09-07T09:11:22.1342895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1343390Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1343532Z 
2025-09-07T09:11:22.1343649Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1344183Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1344690Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1345058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1345507Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1345941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1346365Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1346785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1347255Z     self_outputs = self.self(
2025-09-07T09:11:22.1347670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1348084Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1348222Z 
2025-09-07T09:11:22.1348358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1348883Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1349375Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1349714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1350071Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1350493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1350910Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1351321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1351733Z     self_outputs = self.self(
2025-09-07T09:11:22.1352119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1352542Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1352684Z 
2025-09-07T09:11:22.1352777Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1353288Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1353844Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1354172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1354670Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1355092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1355528Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1355941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1356346Z     self_outputs = self.self(
2025-09-07T09:11:22.1356739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1357186Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1357732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1358349Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1358602Z 
2025-09-07T09:11:22.1358713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1359225Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1359707Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1360080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1360457Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1360863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1361284Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1361694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1362104Z     self_outputs = self.self(
2025-09-07T09:11:22.1362497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1362934Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1363435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1364023Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1364262Z 
2025-09-07T09:11:22.1364350Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1364555Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1364755Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1364948Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1365174Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1365686Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1366163Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1366500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1366857Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1367277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1367698Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1368110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1368592Z     self_outputs = self.self(
2025-09-07T09:11:22.1368986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1369442Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1369969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1370528Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1371065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1371600Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1371805Z 
2025-09-07T09:11:22.1371893Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1372170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1372696Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1373202Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1373549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1373909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1374373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1374796Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1375207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1375638Z     self_outputs = self.self(
2025-09-07T09:11:22.1376050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1376472Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1376597Z 
2025-09-07T09:11:22.1376705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1377209Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1377695Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1378112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1378475Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1378892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1379302Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1379712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1380126Z     self_outputs = self.self(
2025-09-07T09:11:22.1380523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1380950Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1381072Z 
2025-09-07T09:11:22.1381164Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1381696Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1382187Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1382525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1382938Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1383382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1383794Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1384207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1384633Z     self_outputs = self.self(
2025-09-07T09:11:22.1385016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1385477Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1386019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1386653Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1387092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1387436Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1387594Z 
2025-09-07T09:11:22.1387690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1388206Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1388693Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1389087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1389450Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1389861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1390280Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1390691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1391098Z     self_outputs = self.self(
2025-09-07T09:11:22.1391487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1391940Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1392471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1393026Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1393548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1394011Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1394348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1394694Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1394842Z 
2025-09-07T09:11:22.1394950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1395478Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1395970Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1396321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1396682Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1397100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1397607Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1398042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1398469Z     self_outputs = self.self(
2025-09-07T09:11:22.1398870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1399333Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1399867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1400435Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1400649Z 
2025-09-07T09:11:22.1400750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1401311Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1401806Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1402141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1402491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1402909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1403583Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1404004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1404419Z     self_outputs = self.self(
2025-09-07T09:11:22.1404818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1405281Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1405815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1406379Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1406584Z 
2025-09-07T09:11:22.1406693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1407206Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1407687Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1408019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1408380Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1408788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1409208Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1409618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1410025Z     self_outputs = self.self(
2025-09-07T09:11:22.1410420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1410940Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1411184Z 
2025-09-07T09:11:22.1411277Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1411790Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1412396Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1412730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1413078Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1413491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1413907Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1414319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1414766Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1415207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1415676Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1415821Z 
2025-09-07T09:11:22.1415916Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1416430Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1416915Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1417244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1417598Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1418105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1418536Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1418945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1419344Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1419763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1420226Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1420673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1421095Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1421229Z 
2025-09-07T09:11:22.1421329Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1421841Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1422329Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1422672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1423026Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1423437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1423860Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1424258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1424651Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1425058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1425508Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1425956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1426488Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1426863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1427204Z     return self.act(input)
2025-09-07T09:11:22.1427317Z 
2025-09-07T09:11:22.1427413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1427923Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1428416Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1428753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1429103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1429553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1429982Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1430384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1430779Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1431186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1431653Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1432153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1432594Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1432732Z 
2025-09-07T09:11:22.1432836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1433360Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1433853Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1434203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1434565Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1434975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1435401Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1435820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1436231Z     self_outputs = self.self(
2025-09-07T09:11:22.1436639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1437054Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1437198Z 
2025-09-07T09:11:22.1437294Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1437805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1438295Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1438638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1438991Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1439404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1439824Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1440235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1440723Z     self_outputs = self.self(
2025-09-07T09:11:22.1441113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1441554Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1442061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1442668Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1442920Z 
2025-09-07T09:11:22.1443023Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1443528Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1444062Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1444402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1444761Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1455250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1455718Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1456182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1456695Z     self_outputs = self.self(
2025-09-07T09:11:22.1457125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1457547Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1457691Z 
2025-09-07T09:11:22.1457894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1458429Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1458936Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1459285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1459658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1460093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1460525Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1460947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1461355Z     self_outputs = self.self(
2025-09-07T09:11:22.1461756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1462211Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1462724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1463327Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1463574Z 
2025-09-07T09:11:22.1463656Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1463873Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1464076Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1464274Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1464501Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1465029Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1465609Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1465964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1466330Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1466744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1467169Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1467592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1468006Z     self_outputs = self.self(
2025-09-07T09:11:22.1468421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1468938Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1469464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1470021Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1470562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1471102Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1471346Z 
2025-09-07T09:11:22.1471426Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1471666Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1472193Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1472695Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1473048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1473412Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1473835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1474261Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1474678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1475091Z     self_outputs = self.self(
2025-09-07T09:11:22.1475484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1475902Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1476027Z 
2025-09-07T09:11:22.1476350Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1476873Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1477368Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1477703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1478081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1478501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1478924Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1479325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1479735Z     self_outputs = self.self(
2025-09-07T09:11:22.1480174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1480652Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1480780Z 
2025-09-07T09:11:22.1480888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1481396Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1481887Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1482225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1482585Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1483000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1483414Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1483857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1484277Z     self_outputs = self.self(
2025-09-07T09:11:22.1484672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1485101Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1485238Z 
2025-09-07T09:11:22.1485336Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1485884Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1486374Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1486715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1487077Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1487486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1487904Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1488312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1488720Z     self_outputs = self.self(
2025-09-07T09:11:22.1489110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1489569Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1490098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1490690Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1491116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1491461Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1491609Z 
2025-09-07T09:11:22.1491714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1492219Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1492709Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1493049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1493408Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1493815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1494299Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1494750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1495162Z     self_outputs = self.self(
2025-09-07T09:11:22.1495558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1496008Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1496538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1497087Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1497596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1498167Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1498498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1498845Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1499001Z 
2025-09-07T09:11:22.1499096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1499612Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1500099Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1500475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1500842Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1501260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1501680Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1502091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1502496Z     self_outputs = self.self(
2025-09-07T09:11:22.1502893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1503525Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1504056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1504626Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1504832Z 
2025-09-07T09:11:22.1504931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1505452Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1505957Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1506292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1506649Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1507059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1507476Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1507884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1508292Z     self_outputs = self.self(
2025-09-07T09:11:22.1508674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1509289Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1509812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1510374Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1510578Z 
2025-09-07T09:11:22.1510684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1511196Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1511677Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1512015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1512378Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1512876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1513301Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1513720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1514133Z     self_outputs = self.self(
2025-09-07T09:11:22.1514533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1515114Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1515359Z 
2025-09-07T09:11:22.1515463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1515982Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1516475Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1516825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1517191Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1517603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1518019Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1518433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1518883Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1519328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1519758Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1519902Z 
2025-09-07T09:11:22.1520000Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1520512Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1521004Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1521343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1521701Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1522116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1522535Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1522935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1523385Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1523837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1524294Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1524733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1525156Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1525292Z 
2025-09-07T09:11:22.1525394Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1525905Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1526391Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1526753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1527122Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1527539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1527966Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1528358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1528758Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1529207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1529667Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1530108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1530555Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1530936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1531281Z     return self.act(input)
2025-09-07T09:11:22.1531389Z 
2025-09-07T09:11:22.1531493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1532006Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1532486Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1532827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1533184Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1533606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1534026Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1534420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1534814Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1535236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1535701Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1536155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1536581Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1536719Z 
2025-09-07T09:11:22.1536816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1537330Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1539253Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1539587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1539946Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1540355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1540773Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1541187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1541605Z     self_outputs = self.self(
2025-09-07T09:11:22.1541999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1542463Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1542599Z 
2025-09-07T09:11:22.1542703Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1543213Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1543694Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1544034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1544390Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1544839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1545264Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1545669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1546079Z     self_outputs = self.self(
2025-09-07T09:11:22.1546478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1546921Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1547425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1548021Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1548273Z 
2025-09-07T09:11:22.1548370Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1548882Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1549370Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1549710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1550078Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1550498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1550917Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1551347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1551762Z     self_outputs = self.self(
2025-09-07T09:11:22.1552168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1552592Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1552722Z 
2025-09-07T09:11:22.1552829Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1553397Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1553919Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1554268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1554632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1555055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1555480Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1555887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1556314Z     self_outputs = self.self(
2025-09-07T09:11:22.1556756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1557224Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1557740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1558328Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1558576Z 
2025-09-07T09:11:22.1558650Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1558844Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1559065Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1559258Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1559478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1559989Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1560473Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1560805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1561156Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1561564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1561968Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1562368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1562764Z     self_outputs = self.self(
2025-09-07T09:11:22.1563140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1563586Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1564084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1564621Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1565140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1565671Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1565893Z 
2025-09-07T09:11:22.1565981Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1566245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1566785Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1567345Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1567676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1568084Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1568511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1568924Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1569331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1569732Z     self_outputs = self.self(
2025-09-07T09:11:22.1570125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1570534Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1570646Z 
2025-09-07T09:11:22.1570781Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1571293Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1571784Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1572107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1572483Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1572886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1573323Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1573725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1574125Z     self_outputs = self.self(
2025-09-07T09:11:22.1574507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1574913Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1575048Z 
2025-09-07T09:11:22.1575138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1575646Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1576143Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1576487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1576837Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1577235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1577637Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1578081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1578481Z     self_outputs = self.self(
2025-09-07T09:11:22.1578862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1579286Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1579431Z 
2025-09-07T09:11:22.1579521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1580053Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1580528Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1580850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1581248Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1581676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1582125Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1582530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1582927Z     self_outputs = self.self(
2025-09-07T09:11:22.1583310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1583759Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1584299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1584930Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1585351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1585704Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1585859Z 
2025-09-07T09:11:22.1585954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1586460Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1586938Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1587310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1587662Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1588074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1588502Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1588899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1589299Z     self_outputs = self.self(
2025-09-07T09:11:22.1589710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1590158Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1590683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1591224Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1591725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1592200Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1592528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1592865Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1593010Z 
2025-09-07T09:11:22.1593115Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1593621Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1594121Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1594453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1594803Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1595211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1595756Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1596199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1596604Z     self_outputs = self.self(
2025-09-07T09:11:22.1596990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1597434Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1597964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1598526Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1598730Z 
2025-09-07T09:11:22.1598834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1599393Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1599880Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1600218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1600571Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1600979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1601395Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1601830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1602247Z     self_outputs = self.self(
2025-09-07T09:11:22.1602637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1603215Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1603740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1604298Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1604506Z 
2025-09-07T09:11:22.1604602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1605112Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1605596Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1605934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1606286Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1606698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1607112Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1607518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1607920Z     self_outputs = self.self(
2025-09-07T09:11:22.1608310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1608835Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1609073Z 
2025-09-07T09:11:22.1609176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1609690Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1610281Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1610671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1611031Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1611441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1611855Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1612262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1612710Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1613153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1613581Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1613773Z 
2025-09-07T09:11:22.1613878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1614385Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1614868Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1615200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1615556Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1616035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1616459Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1616861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1617260Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1617680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1618188Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1618633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1619051Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1619195Z 
2025-09-07T09:11:22.1619293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1619805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1620298Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1620627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1620985Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1621394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1621817Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1622207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1622596Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1623008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1623458Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1623896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1624400Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1624816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1625156Z     return self.act(input)
2025-09-07T09:11:22.1625263Z 
2025-09-07T09:11:22.1625365Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1625873Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1626357Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1626697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1627050Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1627460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1627922Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1628316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1628708Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1629116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1629579Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1630064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1630489Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1630627Z 
2025-09-07T09:11:22.1630722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1631229Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1631716Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1632043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1632394Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1632808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1633227Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1633639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1634040Z     self_outputs = self.self(
2025-09-07T09:11:22.1634433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1634856Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1634989Z 
2025-09-07T09:11:22.1635089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1635595Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1636074Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1636405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1636758Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1637172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1637586Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1637989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1638453Z     self_outputs = self.self(
2025-09-07T09:11:22.1638877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1639317Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1639809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1640409Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1640658Z 
2025-09-07T09:11:22.1640754Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1641256Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1641743Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1642109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1642464Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1642877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1643290Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1643701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1644107Z     self_outputs = self.self(
2025-09-07T09:11:22.1644528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1644949Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1645078Z 
2025-09-07T09:11:22.1645182Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1645703Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1646184Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1646515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1646869Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1647279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1647695Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1648097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1648505Z     self_outputs = self.self(
2025-09-07T09:11:22.1648893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1649333Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1649830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1650408Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1650652Z 
2025-09-07T09:11:22.1650729Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1650932Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1651133Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1651322Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1651544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1652061Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1652600Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1652968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1653319Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1653729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1654143Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1654553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1654963Z     self_outputs = self.self(
2025-09-07T09:11:22.1655343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1655792Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1656341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1656899Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1657432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1658016Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1658227Z 
2025-09-07T09:11:22.1658341Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1658572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1659087Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1659578Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1659909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1660270Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1660681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1661094Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1661493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1661906Z     self_outputs = self.self(
2025-09-07T09:11:22.1662294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1662705Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1662820Z 
2025-09-07T09:11:22.1662920Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1663421Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1663906Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1664239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1664593Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1665001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1665409Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1665812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1666224Z     self_outputs = self.self(
2025-09-07T09:11:22.1666617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1667104Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1667227Z 
2025-09-07T09:11:22.1667320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1667829Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1668309Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1668638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1668993Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1669403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1669816Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1670276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1670691Z     self_outputs = self.self(
2025-09-07T09:11:22.1671075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1671500Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1671640Z 
2025-09-07T09:11:22.1671731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1672274Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1672764Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1673088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1673446Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1673859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1674275Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1674680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1675083Z     self_outputs = self.self(
2025-09-07T09:11:22.1675477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1675934Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1676462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1677047Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1677462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1677811Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1677964Z 
2025-09-07T09:11:22.1678061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1678573Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1679055Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1679390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1679745Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1680154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1680620Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1681029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1681474Z     self_outputs = self.self(
2025-09-07T09:11:22.1681864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1682322Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1682851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1683392Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1683904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1684373Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1684748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1685110Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1685263Z 
2025-09-07T09:11:22.1685367Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1685893Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1686384Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1686773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1687149Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1687568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1687998Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1688418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1688842Z     self_outputs = self.self(
2025-09-07T09:11:22.1689238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1689696Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1690236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1690805Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1691009Z 
2025-09-07T09:11:22.1691116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1691634Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1692121Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1692458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1692818Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1693238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1693658Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1694069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1694481Z     self_outputs = self.self(
2025-09-07T09:11:22.1694881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1695408Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1695963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1696526Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1696731Z 
2025-09-07T09:11:22.1696831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1697342Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1697892Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1698230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1698583Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1699056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1699485Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1699899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1700306Z     self_outputs = self.self(
2025-09-07T09:11:22.1700700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1701263Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1701506Z 
2025-09-07T09:11:22.1701611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1702120Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1702605Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1702938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1703431Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1703856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1704276Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1704690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1705152Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1705600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1706025Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1706164Z 
2025-09-07T09:11:22.1706273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1706780Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1707268Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1707611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1707970Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1708419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1708845Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1709251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1709738Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1710160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1710671Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1711111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1711539Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1711677Z 
2025-09-07T09:11:22.1711778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1712298Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1712792Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1713128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1713539Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1713957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1714383Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1714773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1715171Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1715640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1716107Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1716550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1716999Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1717381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1717727Z     return self.act(input)
2025-09-07T09:11:22.1717834Z 
2025-09-07T09:11:22.1717939Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1718454Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1718934Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1719274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1719633Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1720052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1720488Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1720888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1721284Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1721701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1722170Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1722625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1723051Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1723192Z 
2025-09-07T09:11:22.1723290Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1723801Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1724390Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1724732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1725093Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1725506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1725928Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1726342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1726748Z     self_outputs = self.self(
2025-09-07T09:11:22.1727143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1727566Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1727733Z 
2025-09-07T09:11:22.1727839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1728348Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1728823Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1729154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1729507Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1729945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1730366Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1730768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1731183Z     self_outputs = self.self(
2025-09-07T09:11:22.1731576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1732018Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1732512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1733112Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1733364Z 
2025-09-07T09:11:22.1733465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1733982Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1734468Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1734804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1735161Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1735575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1735989Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1736402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1736803Z     self_outputs = self.self(
2025-09-07T09:11:22.1737195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1737613Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1737785Z 
2025-09-07T09:11:22.1737889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1738439Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1738958Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1739291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1739643Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1740052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1740462Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1740863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1741269Z     self_outputs = self.self(
2025-09-07T09:11:22.1741706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1742158Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1742659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1743237Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1743488Z 
2025-09-07T09:11:22.1743565Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1743769Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1744004Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1744197Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1744420Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1744931Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1745422Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1745757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1746109Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1746519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1746933Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1747340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1747745Z     self_outputs = self.self(
2025-09-07T09:11:22.1748127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1748576Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1749088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1749639Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1750186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1750713Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1750922Z 
2025-09-07T09:11:22.1750998Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1751226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1751737Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1752223Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1752602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1752998Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1753412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1753830Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1754232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1754641Z     self_outputs = self.self(
2025-09-07T09:11:22.1755034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1755445Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1755561Z 
2025-09-07T09:11:22.1755664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1756206Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1756699Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1757040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1757395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1757807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1758248Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1758663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1759071Z     self_outputs = self.self(
2025-09-07T09:11:22.1759463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1759884Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1760008Z 
2025-09-07T09:11:22.1767379Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1768006Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1768508Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1768870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1769238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1769666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1770090Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1770536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1770957Z     self_outputs = self.self(
2025-09-07T09:11:22.1771353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1771772Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1771911Z 
2025-09-07T09:11:22.1772014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1772533Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1773019Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1773357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1773710Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1774210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1774666Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1775070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1775471Z     self_outputs = self.self(
2025-09-07T09:11:22.1775854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1776303Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1776825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1777407Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1777935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1778282Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1778430Z 
2025-09-07T09:11:22.1778529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1779038Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1779519Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1779889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1780246Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1780654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1781068Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1781472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1781878Z     self_outputs = self.self(
2025-09-07T09:11:22.1782261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1782706Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1783227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1783768Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1784275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1784732Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1785055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1785393Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1785539Z 
2025-09-07T09:11:22.1785640Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1786148Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1786632Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1786972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1787047Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1787335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1787466Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1787752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1787852Z     self_outputs = self.self(
2025-09-07T09:11:22.1788138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1788243Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1788600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1788752Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1788756Z 
2025-09-07T09:11:22.1788849Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1789236Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1789306Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1789530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1789602Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1789881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1789954Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1790262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1790333Z     self_outputs = self.self(
2025-09-07T09:11:22.1790614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1790720Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1791078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1791215Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1791218Z 
2025-09-07T09:11:22.1791312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1791664Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1791734Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1791946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1792013Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1792298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1792366Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1792648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1792712Z     self_outputs = self.self(
2025-09-07T09:11:22.1792995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1793177Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1793180Z 
2025-09-07T09:11:22.1793271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1793626Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1793728Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1793980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1794051Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1794335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1794406Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1794680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1794789Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1795066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1795148Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1795153Z 
2025-09-07T09:11:22.1795278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1795630Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1795699Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1795911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1795987Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1796298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1796378Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1796644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1796716Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1797006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1797111Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1797394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1797467Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1797470Z 
2025-09-07T09:11:22.1797565Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1797922Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1797985Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1798197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1798267Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1798548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1798619Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1798880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1798951Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1799232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1799333Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1799608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1799716Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1799981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1800078Z     return self.act(input)
2025-09-07T09:11:22.1800081Z 
2025-09-07T09:11:22.1800177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1800529Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1800595Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1800808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1800877Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1801162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1801234Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1801530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1801599Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1801883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1802007Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1802285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1802451Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1802455Z 
2025-09-07T09:11:22.1802546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1802904Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1802970Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1803375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1803457Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1803741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1803818Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1804100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1804167Z     self_outputs = self.self(
2025-09-07T09:11:22.1804449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1804524Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1804530Z 
2025-09-07T09:11:22.1804635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1804991Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1805063Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1805284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1805356Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1805641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1805708Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1805988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1806139Z     self_outputs = self.self(
2025-09-07T09:11:22.1806425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1806574Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1806919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1807113Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1807116Z 
2025-09-07T09:11:22.1807212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1807571Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1807636Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1807912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1807988Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1808269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1808343Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1808619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1808687Z     self_outputs = self.self(
2025-09-07T09:11:22.1809032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1809103Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1809106Z 
2025-09-07T09:11:22.1809200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1809553Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1809620Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1809840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1809910Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1810191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1810259Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1810541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1810602Z     self_outputs = self.self(
2025-09-07T09:11:22.1810880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1810978Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1811321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1811499Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1811502Z 
2025-09-07T09:11:22.1811578Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1811647Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1811719Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1811789Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1811887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1812236Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1812345Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1812564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1812669Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1812951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1813018Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1813299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1813364Z     self_outputs = self.self(
2025-09-07T09:11:22.1813639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1813744Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1814122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1814272Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1814596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1814742Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1814745Z 
2025-09-07T09:11:22.1814820Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1814946Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1815311Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1815380Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1815604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1815677Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1815959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1816035Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1816314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1816381Z     self_outputs = self.self(
2025-09-07T09:11:22.1816663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1816742Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1816745Z 
2025-09-07T09:11:22.1816837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1817189Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1817263Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1817479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1817553Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1817865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1817934Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1818221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1818285Z     self_outputs = self.self(
2025-09-07T09:11:22.1818568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1818686Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1818718Z 
2025-09-07T09:11:22.1818816Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1819167Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1819232Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1819453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1819521Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1819807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1819876Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1820189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1820263Z     self_outputs = self.self(
2025-09-07T09:11:22.1820539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1820624Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1820627Z 
2025-09-07T09:11:22.1820720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1821077Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1821176Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1821396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1821480Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1821764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1821845Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1822120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1822184Z     self_outputs = self.self(
2025-09-07T09:11:22.1822464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1822575Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1822937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1823105Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1823300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1823393Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1823397Z 
2025-09-07T09:11:22.1823492Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1823851Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1823917Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1824137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1824208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1824492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1824563Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1824879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1824981Z     self_outputs = self.self(
2025-09-07T09:11:22.1825258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1825368Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1825722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1825857Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1826184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1826267Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1826495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1826586Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1826589Z 
2025-09-07T09:11:22.1826687Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1827045Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1827109Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1827367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1827438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1827722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1827790Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1828078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1828143Z     self_outputs = self.self(
2025-09-07T09:11:22.1828419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1828526Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1828879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1829031Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1829034Z 
2025-09-07T09:11:22.1829126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1829484Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1829550Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1829763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1829837Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1830114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1830188Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1830466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1830527Z     self_outputs = self.self(
2025-09-07T09:11:22.1830807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1830947Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1831303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1831471Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1831475Z 
2025-09-07T09:11:22.1831572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1831924Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1831990Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1832210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1832280Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1832608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1832679Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1832958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1833027Z     self_outputs = self.self(
2025-09-07T09:11:22.1833302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1833485Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1833520Z 
2025-09-07T09:11:22.1833612Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1833969Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1834033Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1834247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1834322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1834600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1834673Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1834953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1835065Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1835349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1835425Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1835428Z 
2025-09-07T09:11:22.1835526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1835885Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1835948Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1836160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1836230Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1836509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1836590Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1836852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1836929Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1837257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1837395Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1837679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1837754Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1837757Z 
2025-09-07T09:11:22.1837855Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1838206Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1838271Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1838483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1838928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1839213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1839291Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1839554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1839621Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1839903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1840036Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1840318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1840427Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1840640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1840707Z     return self.act(input)
2025-09-07T09:11:22.1840713Z 
2025-09-07T09:11:22.1840806Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1841155Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1841222Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1841434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1841505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1841782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1841854Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1842114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1842180Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1842464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1842576Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1842859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1842933Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1842936Z 
2025-09-07T09:11:22.1843024Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1843371Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1843473Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1843723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1843790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1844064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1844141Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1844418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1844489Z     self_outputs = self.self(
2025-09-07T09:11:22.1844768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1844842Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1844846Z 
2025-09-07T09:11:22.1844965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1845321Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1845389Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1845606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1845679Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1845995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1846065Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1846349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1846414Z     self_outputs = self.self(
2025-09-07T09:11:22.1846697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1846795Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1847143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1847327Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1847330Z 
2025-09-07T09:11:22.1847424Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1847782Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1847848Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1848069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1848141Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1848418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1848487Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1848761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1848826Z     self_outputs = self.self(
2025-09-07T09:11:22.1849103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1849175Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1849178Z 
2025-09-07T09:11:22.1849269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1849620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1849748Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1849961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1850037Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1850313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1850381Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1850656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1850718Z     self_outputs = self.self(
2025-09-07T09:11:22.1850996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1851119Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1851467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1851642Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1851645Z 
2025-09-07T09:11:22.1851725Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1851794Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1851861Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1851964Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1852054Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1852410Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1852474Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1852694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1852768Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1853045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1853116Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1853394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1853457Z     self_outputs = self.self(
2025-09-07T09:11:22.1853737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1853840Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1854186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1854328Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1854654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1854796Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1854799Z 
2025-09-07T09:11:22.1854871Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1854967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1855318Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1855385Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1855598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1855745Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1856029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1856094Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1856372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1856436Z     self_outputs = self.self(
2025-09-07T09:11:22.1856717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1856782Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1856785Z 
2025-09-07T09:11:22.1856875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1857261Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1857328Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1857545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1857613Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1857947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1858017Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1858332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1858401Z     self_outputs = self.self(
2025-09-07T09:11:22.1858678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1858754Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1858759Z 
2025-09-07T09:11:22.1858848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1859195Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1859259Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1859471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1859541Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1859817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1859884Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1860160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1860222Z     self_outputs = self.self(
2025-09-07T09:11:22.1860501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1860578Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1860581Z 
2025-09-07T09:11:22.1860674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1861022Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1861086Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1861301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1861371Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1861652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1861780Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1862059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1862123Z     self_outputs = self.self(
2025-09-07T09:11:22.1862398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1862517Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1862871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1863040Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1863260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1863354Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1863359Z 
2025-09-07T09:11:22.1863448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1863797Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1863861Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1864074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1864190Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1864469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1864537Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1864817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1864880Z     self_outputs = self.self(
2025-09-07T09:11:22.1865157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1865261Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1865634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1865765Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1866081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1866163Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1866350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1866437Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1866440Z 
2025-09-07T09:11:22.1866529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1866880Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1866943Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1867154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1867225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1867500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1867567Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1867886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1867979Z     self_outputs = self.self(
2025-09-07T09:11:22.1868255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1868356Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1868707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1868850Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1868853Z 
2025-09-07T09:11:22.1868943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1869327Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1869390Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1869606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1869674Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1869953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1870018Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1870324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1870390Z     self_outputs = self.self(
2025-09-07T09:11:22.1870667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1870770Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1871124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1871264Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1871267Z 
2025-09-07T09:11:22.1871358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1871706Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1871773Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1871987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1872057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1872336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1872407Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1872682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1872740Z     self_outputs = self.self(
2025-09-07T09:11:22.1873019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1873194Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1873197Z 
2025-09-07T09:11:22.1873293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1873644Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1873712Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1873964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1874066Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1874344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1874409Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1874688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1874790Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1875073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1875148Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1875151Z 
2025-09-07T09:11:22.1875241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1875624Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1875688Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1875902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1875969Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1876247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1876359Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1876625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1876695Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1876981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1877083Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1877360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1877433Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1877436Z 
2025-09-07T09:11:22.1877528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1877877Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1877942Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1878154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1878220Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1878504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1878576Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1878836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1878901Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1879182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1879280Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1879556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1879661Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1879926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1880025Z     return self.act(input)
2025-09-07T09:11:22.1880028Z 
2025-09-07T09:11:22.1880118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1880472Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1880542Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1880753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1880827Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1881103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1881176Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1881468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1881536Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1881823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1881941Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1882222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1882325Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1882328Z 
2025-09-07T09:11:22.1882419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1882772Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1882837Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1883057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1883124Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1883401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1883470Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1883746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1883814Z     self_outputs = self.self(
2025-09-07T09:11:22.1884088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1884165Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1884167Z 
2025-09-07T09:11:22.1884258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1884607Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1884669Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1884888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1884961Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1885237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1885304Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1885583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1885646Z     self_outputs = self.self(
2025-09-07T09:11:22.1885965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1886090Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1886440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1886629Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1886632Z 
2025-09-07T09:11:22.1886729Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1887088Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1887153Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1887373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1887475Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1887761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1887831Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1888133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1888200Z     self_outputs = self.self(
2025-09-07T09:11:22.1888507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1888585Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1888588Z 
2025-09-07T09:11:22.1888678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1889036Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1889104Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1889313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1889387Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1889662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1889734Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1890010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1890070Z     self_outputs = self.self(
2025-09-07T09:11:22.1890349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1890445Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1890792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1890965Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1890968Z 
2025-09-07T09:11:22.1891046Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1891115Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1891181Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1891252Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1891344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1891696Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1891758Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1892013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1892118Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1892396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1892465Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1892740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1892805Z     self_outputs = self.self(
2025-09-07T09:11:22.1893083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1893181Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1893557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1893698Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1894026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1894167Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1894170Z 
2025-09-07T09:11:22.1894243Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1894333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1894716Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1894785Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1895001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1895073Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1895349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1895415Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1895697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1895759Z     self_outputs = self.self(
2025-09-07T09:11:22.1896040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1896105Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1896108Z 
2025-09-07T09:11:22.1896197Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1896552Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1896617Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1896832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1896901Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1897184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1897250Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1897526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1897590Z     self_outputs = self.self(
2025-09-07T09:11:22.1897945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1898055Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1898058Z 
2025-09-07T09:11:22.1898176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1898528Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1898590Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1898802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1898872Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1899151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1899220Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1899495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1899660Z     self_outputs = self.self(
2025-09-07T09:11:22.1899946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1900022Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1900025Z 
2025-09-07T09:11:22.1900125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1900474Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1900545Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1900804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1900874Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1901161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1901228Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1901512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1901575Z     self_outputs = self.self(
2025-09-07T09:11:22.1901849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1901969Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1902326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1902501Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1902689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1902788Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1902793Z 
2025-09-07T09:11:22.1902885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1903364Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1903436Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1903654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1903731Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1904014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1904089Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1904373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1904552Z     self_outputs = self.self(
2025-09-07T09:11:22.1904840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1904950Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1905308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1905437Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1905758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1905842Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1906080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1906177Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1906181Z 
2025-09-07T09:11:22.1906278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1906636Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1906703Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1906918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1907037Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1907319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1907394Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1907673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1907740Z     self_outputs = self.self(
2025-09-07T09:11:22.1908023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1908127Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1908485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1908635Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1908639Z 
2025-09-07T09:11:22.1908737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1909091Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1909161Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1909382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1909449Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1909734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1909803Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1910089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1910149Z     self_outputs = self.self(
2025-09-07T09:11:22.1910426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1910535Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1910934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1911113Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1911116Z 
2025-09-07T09:11:22.1911210Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1911569Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1911635Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1911849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1911928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1912205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1912314Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1912603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1912668Z     self_outputs = self.self(
2025-09-07T09:11:22.1912955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1913135Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1913138Z 
2025-09-07T09:11:22.1913266Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1913620Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1913691Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1913909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1913981Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1914264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1914333Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1914614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1914720Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1914997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1915078Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1915081Z 
2025-09-07T09:11:22.1915171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1915529Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1915594Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1915813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1915879Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1916158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1916241Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1916503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1916579Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1916862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1917033Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1917315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1917389Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1917392Z 
2025-09-07T09:11:22.1917489Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1917840Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1917907Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1918118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1918188Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1918507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1918584Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1918848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1918914Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1919202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1919331Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1919612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1919722Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1919932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1920004Z     return self.act(input)
2025-09-07T09:11:22.1920007Z 
2025-09-07T09:11:22.1920098Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1920456Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1920519Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1920732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1920809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1921087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1921168Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1921427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1921495Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1921781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1921893Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1922178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1922251Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1922256Z 
2025-09-07T09:11:22.1922352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1922701Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1922799Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1923017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1923124Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1923404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1923471Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1923747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1923817Z     self_outputs = self.self(
2025-09-07T09:11:22.1924095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1924174Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1924177Z 
2025-09-07T09:11:22.1924267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1924672Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1924739Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1924955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1925028Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1925305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1925409Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1925689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1925754Z     self_outputs = self.self(
2025-09-07T09:11:22.1926038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1926133Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1926485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1926667Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1926670Z 
2025-09-07T09:11:22.1926766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1927117Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1927186Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1927410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1927483Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1927768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1927835Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1928117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1928180Z     self_outputs = self.self(
2025-09-07T09:11:22.1928459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1928539Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1928542Z 
2025-09-07T09:11:22.1928629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1928984Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1929112Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1929326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1929400Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1929678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1929749Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1930029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1930095Z     self_outputs = self.self(
2025-09-07T09:11:22.1930375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1930464Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1930861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1931049Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1931053Z 
2025-09-07T09:11:22.1931136Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1931205Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1931274Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1931350Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1931478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1931854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1931923Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1932155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1932229Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1932514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1932594Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1932873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1932940Z     self_outputs = self.self(
2025-09-07T09:11:22.1933219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1933321Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1933671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1933814Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1934150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1934294Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1934297Z 
2025-09-07T09:11:22.1934375Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1934468Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1934825Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1934896Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1935112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1935225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1935538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1935609Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1935894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1935959Z     self_outputs = self.self(
2025-09-07T09:11:22.1936240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1936304Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1936308Z 
2025-09-07T09:11:22.1936405Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1936790Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1936858Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1937079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1937147Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1937430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1937498Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1937864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1937928Z     self_outputs = self.self(
2025-09-07T09:11:22.1938216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1938295Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1938302Z 
2025-09-07T09:11:22.1938392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1938751Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1938814Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1939028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1939102Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1939382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1939453Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1939728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1939800Z     self_outputs = self.self(
2025-09-07T09:11:22.1940076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1940153Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1940157Z 
2025-09-07T09:11:22.1940254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1940603Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1940671Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1940886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1940954Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1941234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1941339Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1941656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1941719Z     self_outputs = self.self(
2025-09-07T09:11:22.1942002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1942114Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1942473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1942647Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1942836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1942964Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1942970Z 
2025-09-07T09:11:22.1943062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1943424Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1943488Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1943702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1943774Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1944086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1944158Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1944439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1944503Z     self_outputs = self.self(
2025-09-07T09:11:22.1944786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1944891Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1945250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1945377Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1945703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1945786Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1945967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1946061Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1946066Z 
2025-09-07T09:11:22.1946157Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1946514Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1946578Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1946792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1946867Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1947144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1947216Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1947493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1947634Z     self_outputs = self.self(
2025-09-07T09:11:22.1947908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1948008Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1948363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1948507Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1948511Z 
2025-09-07T09:11:22.1948607Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1948959Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1949058Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1949274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1949342Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1949626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1949694Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1949974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1950066Z     self_outputs = self.self(
2025-09-07T09:11:22.1950342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1950449Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1950803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1950941Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1950944Z 
2025-09-07T09:11:22.1951035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1951388Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1951454Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1951669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1951745Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1952023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1952098Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1952375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1952447Z     self_outputs = self.self(
2025-09-07T09:11:22.1952720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1952900Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1952903Z 
2025-09-07T09:11:22.1953000Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1953353Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1953419Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1953632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1953764Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1954050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1954118Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1954402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1954505Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1954790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1954865Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1954868Z 
2025-09-07T09:11:22.1954957Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1955346Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1955413Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1955634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1955702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1955984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1956092Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1956361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1956437Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1956724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1956834Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1957108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1957184Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1957191Z 
2025-09-07T09:11:22.1957282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1957643Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1957711Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1957925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1958002Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1958285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1958358Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1958621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1958689Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1958974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1959070Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1959349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1959458Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1959669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1959779Z     return self.act(input)
2025-09-07T09:11:22.1959812Z 
2025-09-07T09:11:22.1959907Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1960264Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1960327Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1960538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1960616Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1960895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1960974Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1961260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1961332Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1961621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.1961735Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.1962020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.1962092Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1962096Z 
2025-09-07T09:11:22.1962223Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1962577Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1962642Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1962865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1962935Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1963218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1963287Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1963575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1963639Z     self_outputs = self.self(
2025-09-07T09:11:22.1963915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.1963995Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.1963998Z 
2025-09-07T09:11:22.1964088Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1964450Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1964520Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1964741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1964815Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1965092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1965170Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1965446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1965515Z     self_outputs = self.self(
2025-09-07T09:11:22.1965791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1965952Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1966306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1966488Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1966491Z 
2025-09-07T09:11:22.1966591Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1966943Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1967014Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1967227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1967332Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1967624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1967695Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1967979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1968043Z     self_outputs = self.self(
2025-09-07T09:11:22.1968321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.1968432Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.1968435Z 
2025-09-07T09:11:22.1968526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1968882Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1968951Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1969181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1969252Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1969530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1969607Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1969887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1969955Z     self_outputs = self.self(
2025-09-07T09:11:22.1970230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.1970321Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1970678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1970852Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.1970855Z 
2025-09-07T09:11:22.1970933Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1971005Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1971080Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1971150Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1971241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1971601Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1971670Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1971894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1972041Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1972319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1972394Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1972669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1972737Z     self_outputs = self.self(
2025-09-07T09:11:22.1973015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.1973122Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.1973465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.1973641Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.1973978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.1974118Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.1974122Z 
2025-09-07T09:11:22.1974199Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.1974291Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1974689Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1974754Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1974968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1975048Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1975329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1975405Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1975681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1975746Z     self_outputs = self.self(
2025-09-07T09:11:22.1976030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.1976098Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.1976102Z 
2025-09-07T09:11:22.1976199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1976551Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1976623Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1976842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1976911Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1977194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1977261Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1977546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1977608Z     self_outputs = self.self(
2025-09-07T09:11:22.1977928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.1978007Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.1978045Z 
2025-09-07T09:11:22.1978139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1978536Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1978602Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1978828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1978900Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1979178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1979254Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1979534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1979607Z     self_outputs = self.self(
2025-09-07T09:11:22.1979917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.1979996Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.1979999Z 
2025-09-07T09:11:22.1980096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1980449Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1980521Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1980779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1980856Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1981142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1981217Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1981504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1981570Z     self_outputs = self.self(
2025-09-07T09:11:22.1981850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1981961Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1982330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1982503Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.1982691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1982790Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1982793Z 
2025-09-07T09:11:22.1982887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1983246Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1983309Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1983521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1983598Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1983873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1983946Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1984220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1984331Z     self_outputs = self.self(
2025-09-07T09:11:22.1984645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1984749Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1985110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1985240Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.1985568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.1985651Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.1985833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.1985959Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.1985964Z 
2025-09-07T09:11:22.1986055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1986411Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1986473Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1986692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1986760Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1987073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1987149Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1987431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1987504Z     self_outputs = self.self(
2025-09-07T09:11:22.1987784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1987886Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1988243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1988388Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1988392Z 
2025-09-07T09:11:22.1988489Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1988842Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1988912Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1989126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1989197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1989481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1989549Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1989832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1989896Z     self_outputs = self.self(
2025-09-07T09:11:22.1990179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.1990279Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.1990627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.1990837Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.1990840Z 
2025-09-07T09:11:22.1990935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1991293Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1991356Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1991576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1991644Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1991928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1992035Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1992319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.1992390Z     self_outputs = self.self(
2025-09-07T09:11:22.1992667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.1992844Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.1992854Z 
2025-09-07T09:11:22.1992947Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1993348Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1993419Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1993635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1993713Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1993989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.1994057Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.1994338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.1994440Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.1994728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.1994805Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1994808Z 
2025-09-07T09:11:22.1994902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1995250Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1995317Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1995533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1995599Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1995882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1995956Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1996215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1996289Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1996572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1996717Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1997081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.1997165Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.1997168Z 
2025-09-07T09:11:22.1997260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.1997610Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.1997682Z     layer_outputs = layer_module(
2025-09-07T09:11:22.1997894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.1997970Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.1998277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.1998354Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.1998618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.1998685Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.1998973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.1999069Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.1999387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.1999490Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.1999700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.1999770Z     return self.act(input)
2025-09-07T09:11:22.1999775Z 
2025-09-07T09:11:22.1999867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2000226Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2000290Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2000509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2000578Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2000854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2000935Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2001188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2001265Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2001550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.2001663Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.2001945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.2002019Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2002022Z 
2025-09-07T09:11:22.2002118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2002469Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2002537Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2002790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2002893Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2003305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2003376Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2003662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2003724Z     self_outputs = self.self(
2025-09-07T09:11:22.2004006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.2004088Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.2004091Z 
2025-09-07T09:11:22.2004181Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2004590Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2004657Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2004875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2004943Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2005220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2005295Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2005622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2005692Z     self_outputs = self.self(
2025-09-07T09:11:22.2005970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.2006068Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.2006421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.2006629Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.2006632Z 
2025-09-07T09:11:22.2006733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2007083Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2007154Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2007366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2007435Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2007717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2007784Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2008067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2008130Z     self_outputs = self.self(
2025-09-07T09:11:22.2008411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.2008482Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.2008485Z 
2025-09-07T09:11:22.2008575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2008929Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2009040Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2009314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2009384Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2009668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2009743Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2010019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2010088Z     self_outputs = self.self(
2025-09-07T09:11:22.2010371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.2010473Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.2010848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.2011033Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.2011036Z 
2025-09-07T09:11:22.2011115Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2011185Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2011259Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2011327Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2011425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2011817Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2011883Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2012106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2012176Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2012463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2012528Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2012803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2012871Z     self_outputs = self.self(
2025-09-07T09:11:22.2013149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.2013252Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.2013598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.2013736Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.2014065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.2014206Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.2014209Z 
2025-09-07T09:11:22.2014282Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2014375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2014728Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2014790Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2015002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2015111Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2015390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2015497Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2015778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2015842Z     self_outputs = self.self(
2025-09-07T09:11:22.2016120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.2016184Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.2016187Z 
2025-09-07T09:11:22.2016282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2016630Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2016748Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2016965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2017036Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2017318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2017385Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2017699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2017829Z     self_outputs = self.self(
2025-09-07T09:11:22.2018113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.2018184Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.2018190Z 
2025-09-07T09:11:22.2018285Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2018643Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2018709Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2018924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2018991Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2019270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2019341Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2019617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2019684Z     self_outputs = self.self(
2025-09-07T09:11:22.2019963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.2020044Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.2020046Z 
2025-09-07T09:11:22.2020136Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2020484Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2020552Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2020767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2020841Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2021116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2021220Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2021506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2021601Z     self_outputs = self.self(
2025-09-07T09:11:22.2021882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2021992Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2022354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2022519Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.2022709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.2022805Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.2022841Z 
2025-09-07T09:11:22.2022936Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2023298Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2023361Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2023578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2023647Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2023957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2024031Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2024313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2024384Z     self_outputs = self.self(
2025-09-07T09:11:22.2024659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2024763Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2025124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2025250Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.2025574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.2025656Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.2025846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.2025933Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.2025937Z 
2025-09-07T09:11:22.2026028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2026385Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2026449Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2026668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2026733Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2027010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2027094Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2027370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2027475Z     self_outputs = self.self(
2025-09-07T09:11:22.2027789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2027898Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2028248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2028391Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.2028395Z 
2025-09-07T09:11:22.2028494Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2028843Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2028913Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2029158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2029237Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2033474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2033577Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2033904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2033972Z     self_outputs = self.self(
2025-09-07T09:11:22.2034345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2034464Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2034839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2034995Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.2035000Z 
2025-09-07T09:11:22.2035100Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2035467Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2035534Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2035767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2035849Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2036138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2036213Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2036495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2036567Z     self_outputs = self.self(
2025-09-07T09:11:22.2036849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.2037032Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.2037041Z 
2025-09-07T09:11:22.2037140Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2037501Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2037571Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2037792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2037908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2038223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2038294Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2038576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.2038681Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.2038961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.2039038Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2039042Z 
2025-09-07T09:11:22.2039142Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2039526Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2039594Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2039815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2039886Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2040167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2040241Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2040542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2040615Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2040903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.2041015Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.2041292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.2041370Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2041374Z 
2025-09-07T09:11:22.2041467Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2041818Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2041888Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2042100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2042171Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2042447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2042522Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2042790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2042856Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2043140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.2043234Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.2043517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.2043622Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.2043832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.2043940Z     return self.act(input)
2025-09-07T09:11:22.2043943Z 
2025-09-07T09:11:22.2044085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2044440Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2044504Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2044715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2044788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2045067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2045142Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2045400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2045505Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2045792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.2045907Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.2046190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.2046265Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2046269Z 
2025-09-07T09:11:22.2046397Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2046756Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2046828Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2047044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2047119Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2047399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2047468Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2047753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2047819Z     self_outputs = self.self(
2025-09-07T09:11:22.2048100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward
2025-09-07T09:11:22.2048182Z     query_vectors = self.query(hidden_states)
2025-09-07T09:11:22.2048186Z 
2025-09-07T09:11:22.2048276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2048630Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2048698Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2048913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2048980Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2049254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2049326Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2049603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2049668Z     self_outputs = self.self(
2025-09-07T09:11:22.2049948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.2050080Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.2050461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.2050648Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.2050652Z 
2025-09-07T09:11:22.2050749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2051100Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2051169Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2051380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2051448Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2051765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2051836Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2052119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2052182Z     self_outputs = self.self(
2025-09-07T09:11:22.2052460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward
2025-09-07T09:11:22.2052531Z     key_vectors = self.key(hidden_states)
2025-09-07T09:11:22.2052566Z 
2025-09-07T09:11:22.2052658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2053016Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2053082Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2053301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2053373Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2053655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2053725Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2054001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2054068Z     self_outputs = self.self(
2025-09-07T09:11:22.2054345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward
2025-09-07T09:11:22.2054439Z     attn_scores = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.2054784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.2054959Z     diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key))  # multiply
2025-09-07T09:11:22.2054963Z 
2025-09-07T09:11:22.2055044Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2055113Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2055182Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2055249Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2055339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2055693Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2055756Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2055974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2056082Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2056408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2056475Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2056759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2056825Z     self_outputs = self.self(
2025-09-07T09:11:22.2057101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward
2025-09-07T09:11:22.2057202Z     diagonal_mask = self._sliding_chunks_query_key_matmul(
2025-09-07T09:11:22.2057545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul
2025-09-07T09:11:22.2057719Z     self._mask_invalid_locations(diagonal_attention_scores, window_overlap)
2025-09-07T09:11:22.2058119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations
2025-09-07T09:11:22.2058264Z     input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like(
2025-09-07T09:11:22.2058269Z 
2025-09-07T09:11:22.2058356Z cudagraph partition due to non gpu ops
2025-09-07T09:11:22.2058461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2058863Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2058927Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2059148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2059224Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2059507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2059583Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2059862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2059927Z     self_outputs = self.self(
2025-09-07T09:11:22.2060206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward
2025-09-07T09:11:22.2060273Z     attn_scores += diagonal_mask
2025-09-07T09:11:22.2060277Z 
2025-09-07T09:11:22.2060371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2060719Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2060785Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2060997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2061066Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2061346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2061413Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2061691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2061753Z     self_outputs = self.self(
2025-09-07T09:11:22.2062025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward
2025-09-07T09:11:22.2062096Z     attn_probs = nn.functional.softmax(
2025-09-07T09:11:22.2062100Z 
2025-09-07T09:11:22.2062226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2062579Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2062672Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2062887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2062955Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2063230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2063299Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2063576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2063638Z     self_outputs = self.self(
2025-09-07T09:11:22.2063949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward
2025-09-07T09:11:22.2064031Z     value_vectors = self.value(hidden_states)
2025-09-07T09:11:22.2064034Z 
2025-09-07T09:11:22.2064123Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2064470Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2064535Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2064778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2064850Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2065126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2065192Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2065474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2065538Z     self_outputs = self.self(
2025-09-07T09:11:22.2065813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2065923Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2066282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2066453Z     padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1)
2025-09-07T09:11:22.2066645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.2066738Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.2066742Z 
2025-09-07T09:11:22.2066835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2067192Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2067258Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2067479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2067554Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2067835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2067911Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2068188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2068306Z     self_outputs = self.self(
2025-09-07T09:11:22.2068585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2068727Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2069089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2069217Z     chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs)
2025-09-07T09:11:22.2069547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize
2025-09-07T09:11:22.2069631Z     chunked_hidden_states = nn.functional.pad(
2025-09-07T09:11:22.2069822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad
2025-09-07T09:11:22.2069908Z     return torch._C._nn.pad(input, pad, mode, value)
2025-09-07T09:11:22.2069945Z 
2025-09-07T09:11:22.2070038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2070399Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2070462Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2070684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2070752Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2071063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2071138Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2071416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2071490Z     self_outputs = self.self(
2025-09-07T09:11:22.2071764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2071872Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2072236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2072381Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.2072385Z 
2025-09-07T09:11:22.2072484Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2072833Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2072897Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2073119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2073187Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2073469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2073538Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2073813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2073883Z     self_outputs = self.self(
2025-09-07T09:11:22.2074160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward
2025-09-07T09:11:22.2074268Z     attn_output = self._sliding_chunks_matmul_attn_probs_value(
2025-09-07T09:11:22.2074621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value
2025-09-07T09:11:22.2074829Z     context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value))
2025-09-07T09:11:22.2074833Z 
2025-09-07T09:11:22.2074922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2075278Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2075345Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2075559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2075633Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2075915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2075987Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2076310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward
2025-09-07T09:11:22.2076375Z     self_outputs = self.self(
2025-09-07T09:11:22.2076655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward
2025-09-07T09:11:22.2076835Z     attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous()
2025-09-07T09:11:22.2076838Z 
2025-09-07T09:11:22.2076934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2077512Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2077582Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2077797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2077868Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2078145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward
2025-09-07T09:11:22.2078211Z     self_attn_outputs = self.attention(
2025-09-07T09:11:22.2078492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward
2025-09-07T09:11:22.2078595Z     attn_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:11:22.2078871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward
2025-09-07T09:11:22.2078948Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2078952Z 
2025-09-07T09:11:22.2079040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2079392Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2079456Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2079670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2079738Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2080015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2080090Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2080352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2080426Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2080707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.2080847Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.2081165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward
2025-09-07T09:11:22.2081238Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2081241Z 
2025-09-07T09:11:22.2081335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2081685Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2081749Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2081968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2082036Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2082321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2082425Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2082689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2082757Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2083041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk
2025-09-07T09:11:22.2083137Z     intermediate_output = self.intermediate(attn_output)
2025-09-07T09:11:22.2083447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward
2025-09-07T09:11:22.2083556Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:11:22.2083765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:11:22.2083834Z     return self.act(input)
2025-09-07T09:11:22.2083839Z 
2025-09-07T09:11:22.2083928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:11:22.2084278Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244
2025-09-07T09:11:22.2084343Z     layer_outputs = layer_module(
2025-09-07T09:11:22.2084558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:11:22.2084627Z     return super().__call__(*args, **kwargs)
2025-09-07T09:11:22.2084904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward
2025-09-07T09:11:22.2084978Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:11:22.2085233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:11:22.2085301Z     return forward_fn(*input_tensors)
2025-09-07T09:11:22.2085583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk
2025-09-07T09:11:22.2085697Z     layer_output = self.output(intermediate_output, attn_output)
2025-09-07T09:11:22.2085977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward
2025-09-07T09:11:22.2086047Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:11:22.2086051Z 
2025-09-07T09:12:01.6002155Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:01.6002828Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703
2025-09-07T09:12:01.6003513Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T09:12:01.6003984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward
2025-09-07T09:12:01.6005567Z     x = self.dense(features)
2025-09-07T09:12:01.6005704Z 
2025-09-07T09:12:01.6005809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:01.6006345Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703
2025-09-07T09:12:01.6006884Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T09:12:01.6007340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward
2025-09-07T09:12:01.6007780Z     x = self.decoder(x)
2025-09-07T09:12:01.6007892Z 
2025-09-07T09:12:02.8881625Z 
2025-09-07T09:12:03.1225587Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:12:03.3512438Z running benchmark:   3% 1/30 [00:00<00:06,  4.27it/s]
2025-09-07T09:12:03.5803600Z running benchmark:   7% 2/30 [00:00<00:06,  4.33it/s]
2025-09-07T09:12:03.8119253Z running benchmark:  10% 3/30 [00:00<00:06,  4.35it/s]
2025-09-07T09:12:04.0372199Z running benchmark:  13% 4/30 [00:00<00:05,  4.34it/s]
2025-09-07T09:12:04.2644039Z running benchmark:  17% 5/30 [00:01<00:05,  4.37it/s]
2025-09-07T09:12:04.4958390Z running benchmark:  20% 6/30 [00:01<00:05,  4.38it/s]
2025-09-07T09:12:04.7270232Z running benchmark:  23% 7/30 [00:01<00:05,  4.36it/s]
2025-09-07T09:12:04.9542808Z running benchmark:  27% 8/30 [00:01<00:05,  4.35it/s]
2025-09-07T09:12:05.1814524Z running benchmark:  30% 9/30 [00:02<00:04,  4.37it/s]
2025-09-07T09:12:05.4086382Z running benchmark:  33% 10/30 [00:02<00:04,  4.38it/s]
2025-09-07T09:12:05.7104233Z running benchmark:  37% 11/30 [00:02<00:04,  4.38it/s]
2025-09-07T09:12:05.9402688Z running benchmark:  40% 12/30 [00:02<00:04,  3.99it/s]
2025-09-07T09:12:06.1673391Z running benchmark:  43% 13/30 [00:03<00:04,  4.09it/s]
2025-09-07T09:12:06.3941546Z running benchmark:  47% 14/30 [00:03<00:03,  4.18it/s]
2025-09-07T09:12:06.6215721Z running benchmark:  50% 15/30 [00:03<00:03,  4.25it/s]
2025-09-07T09:12:06.8536039Z running benchmark:  53% 16/30 [00:03<00:03,  4.29it/s]
2025-09-07T09:12:07.0831678Z running benchmark:  57% 17/30 [00:03<00:03,  4.30it/s]
2025-09-07T09:12:07.3099074Z running benchmark:  60% 18/30 [00:04<00:02,  4.31it/s]
2025-09-07T09:12:07.5372681Z running benchmark:  63% 19/30 [00:04<00:02,  4.34it/s]
2025-09-07T09:12:07.7644925Z running benchmark:  67% 20/30 [00:04<00:02,  4.36it/s]
2025-09-07T09:12:07.9953607Z running benchmark:  70% 21/30 [00:04<00:02,  4.37it/s]
2025-09-07T09:12:08.2265580Z running benchmark:  73% 22/30 [00:05<00:01,  4.36it/s]
2025-09-07T09:12:08.4525627Z running benchmark:  77% 23/30 [00:05<00:01,  4.35it/s]
2025-09-07T09:12:08.6789940Z running benchmark:  80% 24/30 [00:05<00:01,  4.37it/s]
2025-09-07T09:12:08.9068555Z running benchmark:  83% 25/30 [00:05<00:01,  4.39it/s]
2025-09-07T09:12:09.1387571Z running benchmark:  87% 26/30 [00:06<00:00,  4.39it/s]
2025-09-07T09:12:09.3716447Z running benchmark:  90% 27/30 [00:06<00:00,  4.36it/s]
2025-09-07T09:12:09.5976015Z running benchmark:  93% 28/30 [00:06<00:00,  4.34it/s]
2025-09-07T09:12:09.8980432Z running benchmark:  97% 29/30 [00:06<00:00,  4.37it/s]
2025-09-07T09:12:09.8980769Z running benchmark: 100% 30/30 [00:07<00:00,  3.99it/s]
2025-09-07T09:12:09.8981026Z running benchmark: 100% 30/30 [00:07<00:00,  4.28it/s]
2025-09-07T09:12:10.0035793Z 1.397x
2025-09-07T09:12:10.0036541Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:12:12.9103919Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:12:12.9104780Z   import pynvml  # type: ignore[import]
2025-09-07T09:12:14.9228561Z 
2025-09-07T09:12:16.2711288Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T09:12:16.2713170Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T09:12:16.5071957Z 
2025-09-07T09:12:16.5072423Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:12:16.5110756Z cpu  eval  hf_Reformer                        
2025-09-07T09:12:19.7238300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:19.7238854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2095, in forward
2025-09-07T09:12:19.7239701Z     least_common_mult_chunk_length = _get_least_common_mult_chunk_len(self.config)
2025-09-07T09:12:19.7240283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 194, in _get_least_common_mult_chunk_len
2025-09-07T09:12:19.7240837Z     return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length)
2025-09-07T09:12:19.7241047Z 
2025-09-07T09:12:19.7241154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:19.7241599Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2099, in forward
2025-09-07T09:12:19.7242131Z     input_shape[-1] % least_common_mult_chunk_length != 0
2025-09-07T09:12:19.7242294Z 
2025-09-07T09:12:24.4189907Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T09:12:24.4190625Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T09:12:24.4191166Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] or:
2025-09-07T09:12:24.4191650Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T09:12:24.4192225Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] to include these operations in the captured graph.
2025-09-07T09:12:24.4192703Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T09:12:24.4193167Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break: from user code at:
2025-09-07T09:12:24.4194018Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2132, in torch_dynamo_resume_in_forward_at_2104
2025-09-07T09:12:24.4194893Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     embedding_output = self.embeddings(
2025-09-07T09:12:24.4195636Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 369, in forward
2025-09-07T09:12:24.4196431Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     position_embeddings = self.position_embeddings(position_ids)
2025-09-07T09:12:24.4197315Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T09:12:24.4198100Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     max_position_id = position_ids.max().item()
2025-09-07T09:12:24.4198956Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T09:12:24.4199326Z W0907 09:12:24.418043 175607 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T09:12:24.5378686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:24.5379231Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 351, in forward
2025-09-07T09:12:24.5379685Z     position_ids = torch.arange(
2025-09-07T09:12:24.5379815Z 
2025-09-07T09:12:24.5379958Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:24.5380405Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 357, in forward
2025-09-07T09:12:24.5380849Z     inputs_embeds = self.word_embeddings(input_ids)
2025-09-07T09:12:24.5381019Z 
2025-09-07T09:12:25.2751293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:25.2751854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T09:12:25.2752322Z     max_position_id = position_ids.max().item()
2025-09-07T09:12:25.2752471Z 
2025-09-07T09:12:25.9044963Z cudagraph partition due to non gpu ops
2025-09-07T09:12:25.9045267Z cudagraph partition due to non gpu ops
2025-09-07T09:12:25.9045534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:25.9046575Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 297, in torch_dynamo_resume_in_forward_at_293
2025-09-07T09:12:25.9047098Z     position_encodings = torch.cat(
2025-09-07T09:12:25.9047241Z 
2025-09-07T09:12:25.9047347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:25.9047886Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 304, in torch_dynamo_resume_in_forward_at_293
2025-09-07T09:12:25.9048373Z     [
2025-09-07T09:12:25.9048742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 305, in <listcomp>
2025-09-07T09:12:25.9049241Z     torch.index_select(position_encodings[i], 0, position_ids[i]).unsqueeze(0)
2025-09-07T09:12:25.9049460Z 
2025-09-07T09:12:26.8214366Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:26.8215005Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 370, in torch_dynamo_resume_in_forward_at_369
2025-09-07T09:12:26.8215521Z     embeddings = embeddings + position_embeddings
2025-09-07T09:12:26.8215680Z 
2025-09-07T09:12:30.4281158Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4281488Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4282101Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4282649Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4283114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4283546Z     layer_outputs = layer(
2025-09-07T09:12:30.4283932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4284346Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4284756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4285185Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4285616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4286484Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4287199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 807, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4287752Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2025-09-07T09:12:30.4287932Z 
2025-09-07T09:12:30.4288043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4288563Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4289081Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4289506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4289994Z     layer_outputs = layer(
2025-09-07T09:12:30.4290381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4290783Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4291178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4291598Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4292016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4292608Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4293220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 807, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4293774Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2025-09-07T09:12:30.4293958Z 
2025-09-07T09:12:30.4294061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4294566Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4295147Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4295558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4295955Z     layer_outputs = layer(
2025-09-07T09:12:30.4296335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4296748Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4297140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4297557Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4298164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4298689Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4299304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 807, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4299851Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2025-09-07T09:12:30.4300024Z 
2025-09-07T09:12:30.4300124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4300627Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4301175Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4301705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4302115Z     layer_outputs = layer(
2025-09-07T09:12:30.4302486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4302891Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4303404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4303834Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4304252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1198, in forward
2025-09-07T09:12:30.4304656Z     query_vectors = self.query(hidden_states)
2025-09-07T09:12:30.4304807Z 
2025-09-07T09:12:30.4304960Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4305464Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4305956Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4306364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4306757Z     layer_outputs = layer(
2025-09-07T09:12:30.4307171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4307581Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4307975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4308399Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4308805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1199, in forward
2025-09-07T09:12:30.4309209Z     key_vectors = self.key(hidden_states)
2025-09-07T09:12:30.4309347Z 
2025-09-07T09:12:30.4309424Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4309627Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4309840Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4310337Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4310826Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4311242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4311642Z     layer_outputs = layer(
2025-09-07T09:12:30.4312014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4312418Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4312811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4313229Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4313654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T09:12:30.4314185Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4314731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4315233Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4315497Z 
2025-09-07T09:12:30.4315587Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4315836Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4316050Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4316561Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4317057Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4317478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4317880Z     layer_outputs = layer(
2025-09-07T09:12:30.4318245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4318652Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4319080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4319508Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4319918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T09:12:30.4320423Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4320956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4321393Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4321558Z 
2025-09-07T09:12:30.4321664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4322177Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4322672Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4323098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4323513Z     layer_outputs = layer(
2025-09-07T09:12:30.4323896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4324300Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4324689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4325118Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4325534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T09:12:30.4326026Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4326237Z 
2025-09-07T09:12:30.4326342Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4326860Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4327357Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4327771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4328170Z     layer_outputs = layer(
2025-09-07T09:12:30.4328555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4328959Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4329350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4329776Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4330242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T09:12:30.4330759Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4330971Z 
2025-09-07T09:12:30.4331071Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4331580Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4332073Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4332493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4332881Z     layer_outputs = layer(
2025-09-07T09:12:30.4333252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4333725Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4334116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4334530Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4334935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1200, in forward
2025-09-07T09:12:30.4335349Z     value_vectors = self.value(hidden_states)
2025-09-07T09:12:30.4335492Z 
2025-09-07T09:12:30.4335569Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4335821Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4336036Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4336537Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4337023Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4337435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4337904Z     layer_outputs = layer(
2025-09-07T09:12:30.4338269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4338671Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4339059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4339477Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4339885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1266, in forward
2025-09-07T09:12:30.4340401Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4340949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4341450Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4341657Z 
2025-09-07T09:12:30.4341737Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4341940Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4342152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4342651Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4343140Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4343550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4343998Z     layer_outputs = layer(
2025-09-07T09:12:30.4344367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4344807Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4345194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4345610Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4346015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T09:12:30.4346460Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4346635Z 
2025-09-07T09:12:30.4346731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4347227Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4347748Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4348159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4348564Z     layer_outputs = layer(
2025-09-07T09:12:30.4348938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4349341Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4349767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4350182Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4350593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T09:12:30.4351032Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4351197Z 
2025-09-07T09:12:30.4351300Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4351798Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4352275Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4352683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4353078Z     layer_outputs = layer(
2025-09-07T09:12:30.4353458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4353854Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4354233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4354651Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4355058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1324, in forward
2025-09-07T09:12:30.4355595Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T09:12:30.4356178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T09:12:30.4356689Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T09:12:30.4356890Z 
2025-09-07T09:12:30.4356985Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4357484Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4357973Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4358458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4358889Z     layer_outputs = layer(
2025-09-07T09:12:30.4359264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4359665Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4360055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T09:12:30.4360536Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T09:12:30.4360996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T09:12:30.4361407Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4361544Z 
2025-09-07T09:12:30.4361638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4362171Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4362668Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4363071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4363470Z     layer_outputs = layer(
2025-09-07T09:12:30.4363847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4364334Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4364778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4365189Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4365586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4365988Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4366402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4366823Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4367228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T09:12:30.4367634Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4367765Z 
2025-09-07T09:12:30.4367869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4368368Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4368851Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4369257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4369657Z     layer_outputs = layer(
2025-09-07T09:12:30.4370054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4370501Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4370939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4371342Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4371729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4372125Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4372577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4373045Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4373449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T09:12:30.4373858Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T09:12:30.4373999Z 
2025-09-07T09:12:30.4374105Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4374603Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4375093Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4375505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4375905Z     layer_outputs = layer(
2025-09-07T09:12:30.4376317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4376767Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4377215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4377620Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4378047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4378474Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4378881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T09:12:30.4379301Z     return self.output(hidden_states)
2025-09-07T09:12:30.4379698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T09:12:30.4380117Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4380252Z 
2025-09-07T09:12:30.4380361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4380854Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4381344Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4381757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4382155Z     layer_outputs = layer(
2025-09-07T09:12:30.4382529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4382930Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4383323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4383747Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4384167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 540, in forward
2025-09-07T09:12:30.4384586Z     query_key_vectors = self.query_key(hidden_states)
2025-09-07T09:12:30.4384742Z 
2025-09-07T09:12:30.4384835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4385337Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4385824Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4386238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4386682Z     layer_outputs = layer(
2025-09-07T09:12:30.4387059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4387516Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4387907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4388324Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4388726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4389209Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4389707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 751, in _hash_vectors
2025-09-07T09:12:30.4390212Z     rotated_vectors = torch.einsum("bmtd,mdhr->bmhtr", vectors, random_rotations)
2025-09-07T09:12:30.4390461Z 
2025-09-07T09:12:30.4390576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4391087Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4391590Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4392020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4392430Z     layer_outputs = layer(
2025-09-07T09:12:30.4392849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4393254Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4393651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4394081Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4394500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4394977Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4395463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 754, in _hash_vectors
2025-09-07T09:12:30.4395950Z     rotated_vectors = torch.cat([rotated_vectors, -rotated_vectors], dim=-1)
2025-09-07T09:12:30.4396154Z 
2025-09-07T09:12:30.4396258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4396760Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4397253Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4397664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4398062Z     layer_outputs = layer(
2025-09-07T09:12:30.4398437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4398844Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4399234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4399646Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4400063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4400533Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4401024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 755, in _hash_vectors
2025-09-07T09:12:30.4401537Z     buckets = torch.argmax(rotated_vectors, dim=-1)
2025-09-07T09:12:30.4401689Z 
2025-09-07T09:12:30.4401789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4402293Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4402788Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4403375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4403786Z     layer_outputs = layer(
2025-09-07T09:12:30.4404166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4404572Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4405054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4405487Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4405898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4406424Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4407090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4407624Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T09:12:30.4408077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 182, in _stable_argsort
2025-09-07T09:12:30.4408580Z     scaled_vector = vector.shape[dim] * vector + (scale_offset % vector.shape[dim])
2025-09-07T09:12:30.4408799Z 
2025-09-07T09:12:30.4408882Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4409120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4409629Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4410127Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4410544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4410946Z     layer_outputs = layer(
2025-09-07T09:12:30.4411325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4411727Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4412120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4412534Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4412948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4413466Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4414079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4414603Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T09:12:30.4415043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 183, in _stable_argsort
2025-09-07T09:12:30.4415483Z     return torch.argsort(scaled_vector, dim=dim)
2025-09-07T09:12:30.4415686Z 
2025-09-07T09:12:30.4415764Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4416011Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4416237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4416734Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4417229Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4417642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4418107Z     layer_outputs = layer(
2025-09-07T09:12:30.4418482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4418887Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4420066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4420500Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4420916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4421350Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4421775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T09:12:30.4422323Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4422920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4423426Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4423631Z 
2025-09-07T09:12:30.4423710Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4423919Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4424150Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4424654Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4425149Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4425556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4425956Z     layer_outputs = layer(
2025-09-07T09:12:30.4426335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4426738Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4427133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4427544Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4427957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4428388Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4428810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T09:12:30.4429340Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4429888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4430311Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4430441Z 
2025-09-07T09:12:30.4430516Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4430801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4431327Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4431820Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4432234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4432630Z     layer_outputs = layer(
2025-09-07T09:12:30.4433009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4433487Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4433878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4434301Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4434811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 541, in forward
2025-09-07T09:12:30.4435235Z     value_vectors = self.value(hidden_states)
2025-09-07T09:12:30.4435369Z 
2025-09-07T09:12:30.4435464Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4435958Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4436443Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4436890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4437291Z     layer_outputs = layer(
2025-09-07T09:12:30.4437660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4438060Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4438446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4438864Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4439273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4439691Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4440108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T09:12:30.4440619Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4441154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4441648Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4441852Z 
2025-09-07T09:12:30.4441927Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4442151Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4442650Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4443136Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4443543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4443933Z     layer_outputs = layer(
2025-09-07T09:12:30.4444304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4444706Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4445091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4445631Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4446043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4446468Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4446887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T09:12:30.4447392Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4447907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4448325Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4457081Z 
2025-09-07T09:12:30.4457274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4458014Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4458563Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4459038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4459466Z     layer_outputs = layer(
2025-09-07T09:12:30.4459863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4460322Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4460737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4461170Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4461619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 595, in forward
2025-09-07T09:12:30.4462167Z     query_key_vectors = self._gather_by_expansion(query_key_vectors, sorted_bucket_idx_per_hash, num_hashes)
2025-09-07T09:12:30.4462746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1099, in _gather_by_expansion
2025-09-07T09:12:30.4463208Z     return torch.gather(vectors, 2, expanded_idxs)
2025-09-07T09:12:30.4463365Z 
2025-09-07T09:12:30.4463473Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4463994Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4464524Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4464972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4465505Z     layer_outputs = layer(
2025-09-07T09:12:30.4465905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4466318Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4466721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4467154Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4467570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4468008Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4468438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 849, in _attend
2025-09-07T09:12:30.4468926Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4469230Z 
2025-09-07T09:12:30.4469344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4469855Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4470359Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4470783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4471188Z     layer_outputs = layer(
2025-09-07T09:12:30.4471562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4471971Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4472370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4472832Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4473251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4473675Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4474099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 845, in _attend
2025-09-07T09:12:30.4474627Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4475210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4475717Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4475922Z 
2025-09-07T09:12:30.4476003Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4476249Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4476748Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4477246Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4477662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4478056Z     layer_outputs = layer(
2025-09-07T09:12:30.4478436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4478840Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4479235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4479657Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4480066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4480499Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4480920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T09:12:30.4481367Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4481535Z 
2025-09-07T09:12:30.4481641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4482135Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4482625Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4483038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4483487Z     layer_outputs = layer(
2025-09-07T09:12:30.4483897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4484298Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4484689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4485108Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4485522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4485940Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4486362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T09:12:30.4486799Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4487000Z 
2025-09-07T09:12:30.4487089Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4487315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4487805Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4488297Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4488706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4489141Z     layer_outputs = layer(
2025-09-07T09:12:30.4489513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4489910Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4490296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4490719Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4491125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 692, in forward
2025-09-07T09:12:30.4491663Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T09:12:30.4492241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T09:12:30.4492770Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T09:12:30.4492978Z 
2025-09-07T09:12:30.4493078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4493585Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4494086Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4494501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4494903Z     layer_outputs = layer(
2025-09-07T09:12:30.4495285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4495687Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4496087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T09:12:30.4496555Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T09:12:30.4497029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T09:12:30.4497448Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4497625Z 
2025-09-07T09:12:30.4497766Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4498299Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4498785Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4499195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4499596Z     layer_outputs = layer(
2025-09-07T09:12:30.4499980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4500429Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4500879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4501335Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4501737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4502140Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4502560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4502998Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4503681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T09:12:30.4504103Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4504236Z 
2025-09-07T09:12:30.4504341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4504855Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4505353Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4505770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4506169Z     layer_outputs = layer(
2025-09-07T09:12:30.4506540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4506994Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4507446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4507856Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4508243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4508644Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4509055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4509487Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4509889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T09:12:30.4510299Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T09:12:30.4510443Z 
2025-09-07T09:12:30.4510542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4511046Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4511539Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4511954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4512469Z     layer_outputs = layer(
2025-09-07T09:12:30.4512847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4513299Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4513749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4514154Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4514542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4514935Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4515347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T09:12:30.4515778Z     return self.output(hidden_states)
2025-09-07T09:12:30.4516296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T09:12:30.4516723Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4516867Z 
2025-09-07T09:12:30.4516965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4517465Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4517958Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4518402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4518808Z     layer_outputs = layer(
2025-09-07T09:12:30.4519187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4519594Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4519989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4520404Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4520819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1198, in forward
2025-09-07T09:12:30.4521228Z     query_vectors = self.query(hidden_states)
2025-09-07T09:12:30.4521362Z 
2025-09-07T09:12:30.4521468Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4521968Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4522451Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4522860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4523257Z     layer_outputs = layer(
2025-09-07T09:12:30.4523630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4524033Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4524417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4524831Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4525242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1199, in forward
2025-09-07T09:12:30.4525647Z     key_vectors = self.key(hidden_states)
2025-09-07T09:12:30.4525775Z 
2025-09-07T09:12:30.4525856Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4526051Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4526314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4526846Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4527337Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4527736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4528131Z     layer_outputs = layer(
2025-09-07T09:12:30.4528506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4528906Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4529288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4529696Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4530140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T09:12:30.4530661Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4531197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4531694Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4531899Z 
2025-09-07T09:12:30.4532050Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4532287Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4532508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4533008Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4533501Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4533906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4534303Z     layer_outputs = layer(
2025-09-07T09:12:30.4534674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4535064Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4535449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4535864Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4536265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T09:12:30.4536766Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4537292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4537779Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4537901Z 
2025-09-07T09:12:30.4538097Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4538590Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4539070Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4539467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4539857Z     layer_outputs = layer(
2025-09-07T09:12:30.4540225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4540696Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4541081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4541538Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4541939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T09:12:30.4542407Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4542608Z 
2025-09-07T09:12:30.4542700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4543190Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4543673Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4544114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4544516Z     layer_outputs = layer(
2025-09-07T09:12:30.4544882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4545276Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4545658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4546071Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4546508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T09:12:30.4546977Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4547179Z 
2025-09-07T09:12:30.4547274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4547764Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4548256Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4548661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4549049Z     layer_outputs = layer(
2025-09-07T09:12:30.4549411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4549806Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4550187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4550592Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4550994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1200, in forward
2025-09-07T09:12:30.4551398Z     value_vectors = self.value(hidden_states)
2025-09-07T09:12:30.4551529Z 
2025-09-07T09:12:30.4551605Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4551801Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4552012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4552510Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4552994Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4553398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4553794Z     layer_outputs = layer(
2025-09-07T09:12:30.4554157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4554596Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4555010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4555419Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4555819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1266, in forward
2025-09-07T09:12:30.4556335Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4556872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4557360Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4557561Z 
2025-09-07T09:12:30.4557636Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4557822Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4558066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4558566Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4559043Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4559443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4559830Z     layer_outputs = layer(
2025-09-07T09:12:30.4560247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4560644Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4561022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4561432Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4561828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T09:12:30.4562267Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4562432Z 
2025-09-07T09:12:30.4562527Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4563014Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4563493Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4563900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4564298Z     layer_outputs = layer(
2025-09-07T09:12:30.4564669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4565069Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4565457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4565864Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4566267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T09:12:30.4566699Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4566867Z 
2025-09-07T09:12:30.4566965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4567454Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4567932Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4568335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4568799Z     layer_outputs = layer(
2025-09-07T09:12:30.4569165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4569557Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4569935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4570345Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4570749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1324, in forward
2025-09-07T09:12:30.4571281Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T09:12:30.4571889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T09:12:30.4572409Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T09:12:30.4572605Z 
2025-09-07T09:12:30.4572698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4573192Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4573675Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4574103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4574498Z     layer_outputs = layer(
2025-09-07T09:12:30.4574861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4575253Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4575634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T09:12:30.4576094Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T09:12:30.4576557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T09:12:30.4576960Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4577090Z 
2025-09-07T09:12:30.4577186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4577720Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4578200Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4578603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4578994Z     layer_outputs = layer(
2025-09-07T09:12:30.4579359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4579796Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4580228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4580624Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4581009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4581393Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4581796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4582213Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4582658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T09:12:30.4583098Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4583223Z 
2025-09-07T09:12:30.4583319Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4583812Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4584292Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4584696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4585093Z     layer_outputs = layer(
2025-09-07T09:12:30.4585460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4585935Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4586381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4586784Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4587168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4587558Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4587994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4588424Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4588822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T09:12:30.4589230Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T09:12:30.4589359Z 
2025-09-07T09:12:30.4589459Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4589942Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4590417Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4590813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4591200Z     layer_outputs = layer(
2025-09-07T09:12:30.4591566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4591998Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4592427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4592827Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4593201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4593587Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4593981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T09:12:30.4594395Z     return self.output(hidden_states)
2025-09-07T09:12:30.4594777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T09:12:30.4595187Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4595318Z 
2025-09-07T09:12:30.4595412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4595904Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4596431Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4596868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4597261Z     layer_outputs = layer(
2025-09-07T09:12:30.4597626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4598020Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4598402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4598813Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4599220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 540, in forward
2025-09-07T09:12:30.4599632Z     query_key_vectors = self.query_key(hidden_states)
2025-09-07T09:12:30.4599813Z 
2025-09-07T09:12:30.4599905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4600402Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4600879Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4601279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4601678Z     layer_outputs = layer(
2025-09-07T09:12:30.4602081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4602487Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4602874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4603438Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4603846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4604313Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4604802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 751, in _hash_vectors
2025-09-07T09:12:30.4605291Z     rotated_vectors = torch.einsum("bmtd,mdhr->bmhtr", vectors, random_rotations)
2025-09-07T09:12:30.4605497Z 
2025-09-07T09:12:30.4605601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4606089Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4606574Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4606986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4607379Z     layer_outputs = layer(
2025-09-07T09:12:30.4607747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4608137Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4608526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4608939Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4609359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4609819Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4610305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 754, in _hash_vectors
2025-09-07T09:12:30.4611225Z     rotated_vectors = torch.cat([rotated_vectors, -rotated_vectors], dim=-1)
2025-09-07T09:12:30.4611429Z 
2025-09-07T09:12:30.4611525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4612016Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4612499Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4612901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4613295Z     layer_outputs = layer(
2025-09-07T09:12:30.4613661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4614054Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4614499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4614924Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4615334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4615800Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4616285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 755, in _hash_vectors
2025-09-07T09:12:30.4616760Z     buckets = torch.argmax(rotated_vectors, dim=-1)
2025-09-07T09:12:30.4616914Z 
2025-09-07T09:12:30.4617013Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4617510Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4618039Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4618451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4618840Z     layer_outputs = layer(
2025-09-07T09:12:30.4619210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4619605Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4619993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4620404Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4620804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4621326Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4621935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4622454Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T09:12:30.4622892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 182, in _stable_argsort
2025-09-07T09:12:30.4623403Z     scaled_vector = vector.shape[dim] * vector + (scale_offset % vector.shape[dim])
2025-09-07T09:12:30.4623614Z 
2025-09-07T09:12:30.4623690Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4623912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4624405Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4624944Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4625349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4625781Z     layer_outputs = layer(
2025-09-07T09:12:30.4626152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4626548Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4626931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4627378Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4627780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4628290Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4628933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4629456Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T09:12:30.4629894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 183, in _stable_argsort
2025-09-07T09:12:30.4630330Z     return torch.argsort(scaled_vector, dim=dim)
2025-09-07T09:12:30.4630475Z 
2025-09-07T09:12:30.4630549Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4630781Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4630992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4631487Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4631979Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4632388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4632784Z     layer_outputs = layer(
2025-09-07T09:12:30.4633149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4633543Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4633932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4634343Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4634749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4635165Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4635587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T09:12:30.4636120Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4636680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4637169Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4637371Z 
2025-09-07T09:12:30.4637443Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4637632Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4637848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4638336Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4638820Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4639267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4639694Z     layer_outputs = layer(
2025-09-07T09:12:30.4640063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4640458Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4640838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4641247Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4641649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4642065Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4642520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T09:12:30.4643048Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4643590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4644018Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4644139Z 
2025-09-07T09:12:30.4644215Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4644433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4644968Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4645452Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4645855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4646247Z     layer_outputs = layer(
2025-09-07T09:12:30.4646609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4646996Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4647376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4647786Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4648192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 541, in forward
2025-09-07T09:12:30.4648594Z     value_vectors = self.value(hidden_states)
2025-09-07T09:12:30.4648727Z 
2025-09-07T09:12:30.4648818Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4649311Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4649803Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4650213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4650607Z     layer_outputs = layer(
2025-09-07T09:12:30.4650977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4651373Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4651762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4652173Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4652575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4653061Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4653514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T09:12:30.4654021Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4654546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4655029Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4655236Z 
2025-09-07T09:12:30.4655311Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4655541Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4656034Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4656564Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4656972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4657365Z     layer_outputs = layer(
2025-09-07T09:12:30.4657796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4658192Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4658573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4659070Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4659484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4659912Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4660332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T09:12:30.4660835Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4661346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4661764Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4661889Z 
2025-09-07T09:12:30.4661983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4662476Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4662959Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4663358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4663753Z     layer_outputs = layer(
2025-09-07T09:12:30.4664123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4664523Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4664903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4665314Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4665712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 595, in forward
2025-09-07T09:12:30.4666244Z     query_key_vectors = self._gather_by_expansion(query_key_vectors, sorted_bucket_idx_per_hash, num_hashes)
2025-09-07T09:12:30.4666815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1099, in _gather_by_expansion
2025-09-07T09:12:30.4667319Z     return torch.gather(vectors, 2, expanded_idxs)
2025-09-07T09:12:30.4667521Z 
2025-09-07T09:12:30.4667613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4668101Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4668577Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4668974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4669357Z     layer_outputs = layer(
2025-09-07T09:12:30.4669724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4670114Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4670532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4670952Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4671350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4671769Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4672184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 849, in _attend
2025-09-07T09:12:30.4672656Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4672859Z 
2025-09-07T09:12:30.4672993Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4673485Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4673968Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4674375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4674766Z     layer_outputs = layer(
2025-09-07T09:12:30.4675128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4675519Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4675899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4676316Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4676716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4677127Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4677538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 845, in _attend
2025-09-07T09:12:30.4678052Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4678585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4679068Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4679269Z 
2025-09-07T09:12:30.4679344Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4679563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4680052Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4680535Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4680936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4681402Z     layer_outputs = layer(
2025-09-07T09:12:30.4681763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4682157Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4682539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4682947Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4683345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4683761Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4684170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T09:12:30.4684647Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4684815Z 
2025-09-07T09:12:30.4684912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4685392Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4685887Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4686295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4686692Z     layer_outputs = layer(
2025-09-07T09:12:30.4687096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4687497Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4687889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4688312Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4688725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4689147Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4689557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T09:12:30.4689991Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4690158Z 
2025-09-07T09:12:30.4690233Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4690454Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4690945Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4691426Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4691834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4692228Z     layer_outputs = layer(
2025-09-07T09:12:30.4692597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4692991Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4693376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4693789Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4694192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 692, in forward
2025-09-07T09:12:30.4694726Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T09:12:30.4695349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T09:12:30.4695901Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T09:12:30.4696103Z 
2025-09-07T09:12:30.4696197Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4696703Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4697189Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4697600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4698057Z     layer_outputs = layer(
2025-09-07T09:12:30.4698443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4698890Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4699287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T09:12:30.4699753Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T09:12:30.4700222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T09:12:30.4700634Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4700769Z 
2025-09-07T09:12:30.4700908Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4701409Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4701895Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4702313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4702715Z     layer_outputs = layer(
2025-09-07T09:12:30.4703227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1601, in forward
2025-09-07T09:12:30.4703653Z     attn_output = prev_attn_output + attn_output
2025-09-07T09:12:30.4703794Z 
2025-09-07T09:12:30.4703888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4704386Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4704873Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4705283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4705676Z     layer_outputs = layer(
2025-09-07T09:12:30.4706048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4706504Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4706952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4707362Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4707757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4708157Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4708583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4709023Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4709444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T09:12:30.4709989Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4710132Z 
2025-09-07T09:12:30.4710234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4710737Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4711224Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4711644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4712035Z     layer_outputs = layer(
2025-09-07T09:12:30.4712413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4712863Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4713370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4713786Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4714172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4714568Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4714978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4715405Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4715860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T09:12:30.4716270Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T09:12:30.4716409Z 
2025-09-07T09:12:30.4716507Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4717013Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4717511Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4717919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4718313Z     layer_outputs = layer(
2025-09-07T09:12:30.4718691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4719142Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4719583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4719987Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4720371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4720763Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4721173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T09:12:30.4721591Z     return self.output(hidden_states)
2025-09-07T09:12:30.4721974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T09:12:30.4722380Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4722385Z 
2025-09-07T09:12:30.4722487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4722833Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4722918Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4723247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4723344Z     layer_outputs = layer(
2025-09-07T09:12:30.4723620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4723733Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4723737Z 
2025-09-07T09:12:30.4723838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4724181Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4724266Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4724532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4724631Z     layer_outputs = layer(
2025-09-07T09:12:30.4724911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4724981Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4725258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4725346Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4725645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1198, in forward
2025-09-07T09:12:30.4725733Z     query_vectors = self.query(hidden_states)
2025-09-07T09:12:30.4725737Z 
2025-09-07T09:12:30.4725829Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4726178Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4726259Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4726534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4726595Z     layer_outputs = layer(
2025-09-07T09:12:30.4726861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4726937Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4727206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4727294Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4727563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1199, in forward
2025-09-07T09:12:30.4727637Z     key_vectors = self.key(hidden_states)
2025-09-07T09:12:30.4727642Z 
2025-09-07T09:12:30.4727726Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4727796Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4727902Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4728242Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4728332Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4728606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4728671Z     layer_outputs = layer(
2025-09-07T09:12:30.4728945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4729015Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4729335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4729526Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4729798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T09:12:30.4729991Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4730281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4730454Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4730458Z 
2025-09-07T09:12:30.4730535Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4730627Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4730726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4731107Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4731199Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4731468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4731539Z     layer_outputs = layer(
2025-09-07T09:12:30.4731805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4731906Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4732184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4732269Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4732544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1265, in forward
2025-09-07T09:12:30.4732719Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4733009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4733080Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4733083Z 
2025-09-07T09:12:30.4733179Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4733530Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4733611Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4733887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4733953Z     layer_outputs = layer(
2025-09-07T09:12:30.4734220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4734297Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4734564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4734652Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4734917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T09:12:30.4735073Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4735076Z 
2025-09-07T09:12:30.4735169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4735510Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4735661Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4735930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4735997Z     layer_outputs = layer(
2025-09-07T09:12:30.4736263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4736336Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4736606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4736687Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4736960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1272, in forward
2025-09-07T09:12:30.4737134Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4737139Z 
2025-09-07T09:12:30.4737238Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4737579Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4737737Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4738007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4738067Z     layer_outputs = layer(
2025-09-07T09:12:30.4738376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4738442Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4738717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4738801Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4739069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1200, in forward
2025-09-07T09:12:30.4739157Z     value_vectors = self.value(hidden_states)
2025-09-07T09:12:30.4739161Z 
2025-09-07T09:12:30.4739237Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4739314Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4739406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4739749Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4739833Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4740098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4740165Z     layer_outputs = layer(
2025-09-07T09:12:30.4740432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4740502Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4740767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4740844Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4741119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1266, in forward
2025-09-07T09:12:30.4741313Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4741607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4741783Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4741816Z 
2025-09-07T09:12:30.4741891Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4741960Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4742052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4742417Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4742494Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4742768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4742828Z     layer_outputs = layer(
2025-09-07T09:12:30.4743104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4743183Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4743482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4743585Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4743861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T09:12:30.4743983Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4743987Z 
2025-09-07T09:12:30.4744076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4744460Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4744541Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4744808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4744877Z     layer_outputs = layer(
2025-09-07T09:12:30.4745141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4745205Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4745473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4745550Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4745820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1308, in forward
2025-09-07T09:12:30.4745922Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4745925Z 
2025-09-07T09:12:30.4746017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4746355Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4746435Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4746701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4746759Z     layer_outputs = layer(
2025-09-07T09:12:30.4747028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4747091Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4747358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4747438Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4747702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1324, in forward
2025-09-07T09:12:30.4747950Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T09:12:30.4748303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T09:12:30.4748444Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T09:12:30.4748447Z 
2025-09-07T09:12:30.4748544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4748893Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4748976Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4749243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4749312Z     layer_outputs = layer(
2025-09-07T09:12:30.4749617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4749694Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4749960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T09:12:30.4750093Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T09:12:30.4750366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T09:12:30.4750477Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4750480Z 
2025-09-07T09:12:30.4750583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4750926Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4751017Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4751285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4751346Z     layer_outputs = layer(
2025-09-07T09:12:30.4751621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4751740Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4752016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4752089Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4752350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4752429Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4752718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4752802Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4753074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T09:12:30.4753163Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4753166Z 
2025-09-07T09:12:30.4753260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4753602Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4753690Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4753956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4754066Z     layer_outputs = layer(
2025-09-07T09:12:30.4754334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4754482Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4754755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4754830Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4755097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4755163Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4755456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4755531Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4755832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T09:12:30.4755919Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T09:12:30.4755922Z 
2025-09-07T09:12:30.4756012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4756355Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4756437Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4761811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4761913Z     layer_outputs = layer(
2025-09-07T09:12:30.4762231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4762371Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4762658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4762735Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4763010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4763078Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4763375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T09:12:30.4763443Z     return self.output(hidden_states)
2025-09-07T09:12:30.4763725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T09:12:30.4763808Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4763814Z 
2025-09-07T09:12:30.4763926Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4764322Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4764414Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4764712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4764780Z     layer_outputs = layer(
2025-09-07T09:12:30.4765067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4765142Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4765411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4765502Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4765823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 540, in forward
2025-09-07T09:12:30.4765954Z     query_key_vectors = self.query_key(hidden_states)
2025-09-07T09:12:30.4765958Z 
2025-09-07T09:12:30.4766057Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4766423Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4766515Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4766793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4766862Z     layer_outputs = layer(
2025-09-07T09:12:30.4767132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4767249Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4767520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4767607Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4767881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4768024Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4768347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 751, in _hash_vectors
2025-09-07T09:12:30.4768498Z     rotated_vectors = torch.einsum("bmtd,mdhr->bmhtr", vectors, random_rotations)
2025-09-07T09:12:30.4768501Z 
2025-09-07T09:12:30.4768600Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4768946Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4769029Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4769303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4769367Z     layer_outputs = layer(
2025-09-07T09:12:30.4769641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4769711Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4769983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4770070Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4770346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4770492Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4770779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 754, in _hash_vectors
2025-09-07T09:12:30.4770918Z     rotated_vectors = torch.cat([rotated_vectors, -rotated_vectors], dim=-1)
2025-09-07T09:12:30.4770922Z 
2025-09-07T09:12:30.4771020Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4771367Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4771450Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4771716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4771780Z     layer_outputs = layer(
2025-09-07T09:12:30.4772083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4772183Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4772452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4772531Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4772802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 578, in forward
2025-09-07T09:12:30.4772934Z     buckets = self._hash_vectors(query_key_vectors, num_hashes, attention_mask)
2025-09-07T09:12:30.4773218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 755, in _hash_vectors
2025-09-07T09:12:30.4773304Z     buckets = torch.argmax(rotated_vectors, dim=-1)
2025-09-07T09:12:30.4773307Z 
2025-09-07T09:12:30.4773448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4773797Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4773877Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4774146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4774207Z     layer_outputs = layer(
2025-09-07T09:12:30.4774508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4774572Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4774835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4774920Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4775189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4775377Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4775739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4775839Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T09:12:30.4776131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 182, in _stable_argsort
2025-09-07T09:12:30.4776276Z     scaled_vector = vector.shape[dim] * vector + (scale_offset % vector.shape[dim])
2025-09-07T09:12:30.4776279Z 
2025-09-07T09:12:30.4776357Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4776449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4776801Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4776882Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4777148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4777211Z     layer_outputs = layer(
2025-09-07T09:12:30.4777478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4777555Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4777897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4777979Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4778249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 587, in forward
2025-09-07T09:12:30.4778501Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2025-09-07T09:12:30.4778861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 796, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2025-09-07T09:12:30.4778954Z     sorted_bucket_idx = _stable_argsort(buckets, dim=-1)
2025-09-07T09:12:30.4779250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 183, in _stable_argsort
2025-09-07T09:12:30.4779336Z     return torch.argsort(scaled_vector, dim=dim)
2025-09-07T09:12:30.4779339Z 
2025-09-07T09:12:30.4779418Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4779487Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4779578Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4779964Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4780047Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4780315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4780375Z     layer_outputs = layer(
2025-09-07T09:12:30.4780643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4780746Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4781015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4781099Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4781373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4781466Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4781741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T09:12:30.4781947Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4782241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4782390Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4782393Z 
2025-09-07T09:12:30.4782471Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4782539Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4782633Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4782982Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4783063Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4783339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4783402Z     layer_outputs = layer(
2025-09-07T09:12:30.4783676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4783744Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4784007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4784090Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4784353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4784509Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4784773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 859, in _attend
2025-09-07T09:12:30.4784972Z     key_value_bucket_idx = self._look_adjacent(query_bucket_idx, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4785257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4785324Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4785327Z 
2025-09-07T09:12:30.4785400Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4785488Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4785865Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4785946Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4786212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4786273Z     layer_outputs = layer(
2025-09-07T09:12:30.4786537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4786605Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4786901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4786981Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4787248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 541, in forward
2025-09-07T09:12:30.4787321Z     value_vectors = self.value(hidden_states)
2025-09-07T09:12:30.4787329Z 
2025-09-07T09:12:30.4787422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4787759Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4787839Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4788102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4788164Z     layer_outputs = layer(
2025-09-07T09:12:30.4788437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4788501Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4788770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4788848Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4789113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4789202Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4789464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T09:12:30.4789639Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4789922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4790058Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4790062Z 
2025-09-07T09:12:30.4790129Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4790254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4790594Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4790704Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4790972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4791031Z     layer_outputs = layer(
2025-09-07T09:12:30.4791296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4791360Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4791623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4791701Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4791999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4792091Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4792353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 844, in _attend
2025-09-07T09:12:30.4792519Z     key_vectors = self._look_adjacent(key_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4792801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 400, in _look_adjacent
2025-09-07T09:12:30.4792895Z     return torch.cat(slices, dim=3)
2025-09-07T09:12:30.4792899Z 
2025-09-07T09:12:30.4792992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4793329Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4793409Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4793673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4793731Z     layer_outputs = layer(
2025-09-07T09:12:30.4793996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4794058Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4794324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4794399Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4794660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 595, in forward
2025-09-07T09:12:30.4794857Z     query_key_vectors = self._gather_by_expansion(query_key_vectors, sorted_bucket_idx_per_hash, num_hashes)
2025-09-07T09:12:30.4795167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1099, in _gather_by_expansion
2025-09-07T09:12:30.4795254Z     return torch.gather(vectors, 2, expanded_idxs)
2025-09-07T09:12:30.4795257Z 
2025-09-07T09:12:30.4795345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4795682Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4795758Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4796024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4796080Z     layer_outputs = layer(
2025-09-07T09:12:30.4796347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4796558Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4796873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4796957Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4797222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4797312Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4797589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 849, in _attend
2025-09-07T09:12:30.4797738Z     query_key_dots = torch.matmul(query_vectors, key_vectors.transpose(-1, -2))
2025-09-07T09:12:30.4797741Z 
2025-09-07T09:12:30.4797846Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4798238Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4798335Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4798618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4798685Z     layer_outputs = layer(
2025-09-07T09:12:30.4798963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4799034Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4799340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4799431Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4799706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4799812Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4800088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 845, in _attend
2025-09-07T09:12:30.4800288Z     value_vectors = self._look_adjacent(value_vectors, self.num_chunks_before, self.num_chunks_after)
2025-09-07T09:12:30.4800579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 399, in _look_adjacent
2025-09-07T09:12:30.4800728Z     slices.append(torch.cat([vectors[:, :, i:, ...], vectors[:, :, :i, ...]], dim=2))
2025-09-07T09:12:30.4800732Z 
2025-09-07T09:12:30.4800808Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4800909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4801259Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4801344Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4801613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4801683Z     layer_outputs = layer(
2025-09-07T09:12:30.4801951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4802024Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4802293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4802379Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4802655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4802786Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4803241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T09:12:30.4803417Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4803421Z 
2025-09-07T09:12:30.4803521Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4803867Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4803946Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4804226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4804290Z     layer_outputs = layer(
2025-09-07T09:12:30.4804609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4804678Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4804953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4805034Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4805301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 637, in forward
2025-09-07T09:12:30.4805395Z     out_vectors, logits, attention_probs = self._attend(
2025-09-07T09:12:30.4805714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 931, in _attend
2025-09-07T09:12:30.4805822Z     out_vectors = torch.matmul(attention_probs, value_vectors)
2025-09-07T09:12:30.4805825Z 
2025-09-07T09:12:30.4805898Z cudagraph partition due to non gpu ops
2025-09-07T09:12:30.4805991Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4806345Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4806424Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4806697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4806757Z     layer_outputs = layer(
2025-09-07T09:12:30.4807028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4807094Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4807359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1417, in forward
2025-09-07T09:12:30.4807440Z     self_attention_outputs = self.self_attention(
2025-09-07T09:12:30.4807705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 692, in forward
2025-09-07T09:12:30.4807917Z     out_vectors = self._merge_hidden_size_dims(out_vectors, self.num_attention_heads, self.attention_head_size)
2025-09-07T09:12:30.4808234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 415, in _merge_hidden_size_dims
2025-09-07T09:12:30.4808375Z     return torch.reshape(x, (x.size()[0], -1, num_attn_heads * attn_head_size))
2025-09-07T09:12:30.4808384Z 
2025-09-07T09:12:30.4808475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4808821Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4808910Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4809178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4809292Z     layer_outputs = layer(
2025-09-07T09:12:30.4809594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1587, in forward
2025-09-07T09:12:30.4809662Z     attn_outputs = self.attention(
2025-09-07T09:12:30.4809932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1457, in forward
2025-09-07T09:12:30.4810066Z     attention_output = self.output(self_attention_outputs.hidden_states)
2025-09-07T09:12:30.4810339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1371, in forward
2025-09-07T09:12:30.4810416Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4810419Z 
2025-09-07T09:12:30.4810512Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4810884Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4810967Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4811238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4811297Z     layer_outputs = layer(
2025-09-07T09:12:30.4811566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4811690Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4811993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4812066Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4812332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4812410Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4812701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4812776Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4813043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1479, in forward
2025-09-07T09:12:30.4813114Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4813117Z 
2025-09-07T09:12:30.4813214Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4813554Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4813635Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4813910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4813977Z     layer_outputs = layer(
2025-09-07T09:12:30.4814239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4814349Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4814617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4814683Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4814944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4815012Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4815302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1518, in forward_chunk
2025-09-07T09:12:30.4815416Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4815715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1481, in forward
2025-09-07T09:12:30.4815794Z     hidden_states = self.act_fn(hidden_states)
2025-09-07T09:12:30.4815798Z 
2025-09-07T09:12:30.4815889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4816233Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4816311Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4816576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4816639Z     layer_outputs = layer(
2025-09-07T09:12:30.4816939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4817057Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4817324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1509, in forward
2025-09-07T09:12:30.4817389Z     return apply_chunking_to_forward(
2025-09-07T09:12:30.4817651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:30.4817763Z     return forward_fn(*input_tensors)
2025-09-07T09:12:30.4818088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1519, in forward_chunk
2025-09-07T09:12:30.4818155Z     return self.output(hidden_states)
2025-09-07T09:12:30.4818425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1493, in forward
2025-09-07T09:12:30.4818502Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:12:30.4818506Z 
2025-09-07T09:12:30.4818596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4818940Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4819018Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4819286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1722, in forward
2025-09-07T09:12:30.4819347Z     layer_outputs = layer(
2025-09-07T09:12:30.4819615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1612, in forward
2025-09-07T09:12:30.4819725Z     hidden_states = hidden_states + self.feed_forward(attn_output)
2025-09-07T09:12:30.4819728Z 
2025-09-07T09:12:30.4819815Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:30.4820156Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1841, in torch_dynamo_resume_in_forward_at_1837
2025-09-07T09:12:30.4820235Z     hidden_states = _ReversibleFunction.apply(
2025-09-07T09:12:30.4820506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1753, in forward
2025-09-07T09:12:30.4820609Z     return torch.cat([attn_output, hidden_states], dim=-1)
2025-09-07T09:12:30.4820612Z 
2025-09-07T09:12:44.9421252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:12:44.9421913Z    File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2495, in torch_dynamo_resume_in_forward_at_2481
2025-09-07T09:12:44.9422425Z     logits = self.lm_head(sequence_output)
2025-09-07T09:12:44.9422882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1886, in forward
2025-09-07T09:12:44.9423855Z     return apply_chunking_to_forward(self.forward_chunk, self.chunk_size_lm_head, self.seq_len_dim, hidden_states)
2025-09-07T09:12:44.9424520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:12:44.9424929Z     return forward_fn(*input_tensors)
2025-09-07T09:12:44.9425363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 1889, in forward_chunk
2025-09-07T09:12:44.9425808Z     hidden_states = self.decoder(hidden_states)
2025-09-07T09:12:44.9425955Z 
2025-09-07T09:12:45.7496515Z 
2025-09-07T09:12:46.1707240Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:12:46.5900870Z running benchmark:   3% 1/30 [00:00<00:12,  2.38it/s]
2025-09-07T09:12:47.0057727Z running benchmark:   7% 2/30 [00:00<00:11,  2.38it/s]
2025-09-07T09:12:47.4220096Z running benchmark:  10% 3/30 [00:01<00:11,  2.39it/s]
2025-09-07T09:12:47.8391444Z running benchmark:  13% 4/30 [00:01<00:10,  2.40it/s]
2025-09-07T09:12:48.2555943Z running benchmark:  17% 5/30 [00:02<00:10,  2.40it/s]
2025-09-07T09:12:48.6710948Z running benchmark:  20% 6/30 [00:02<00:10,  2.40it/s]
2025-09-07T09:12:49.0859220Z running benchmark:  23% 7/30 [00:02<00:09,  2.40it/s]
2025-09-07T09:12:49.5029859Z running benchmark:  27% 8/30 [00:03<00:09,  2.40it/s]
2025-09-07T09:12:49.9222468Z running benchmark:  30% 9/30 [00:03<00:08,  2.40it/s]
2025-09-07T09:12:50.3416527Z running benchmark:  33% 10/30 [00:04<00:08,  2.40it/s]
2025-09-07T09:12:50.7589215Z running benchmark:  37% 11/30 [00:04<00:07,  2.39it/s]
2025-09-07T09:12:51.1759919Z running benchmark:  40% 12/30 [00:05<00:07,  2.39it/s]
2025-09-07T09:12:51.5900460Z running benchmark:  43% 13/30 [00:05<00:07,  2.40it/s]
2025-09-07T09:12:52.0065127Z running benchmark:  47% 14/30 [00:05<00:06,  2.40it/s]
2025-09-07T09:12:52.4243141Z running benchmark:  50% 15/30 [00:06<00:06,  2.40it/s]
2025-09-07T09:12:52.8412367Z running benchmark:  53% 16/30 [00:06<00:05,  2.40it/s]
2025-09-07T09:12:53.2575037Z running benchmark:  57% 17/30 [00:07<00:05,  2.40it/s]
2025-09-07T09:12:53.6718931Z running benchmark:  60% 18/30 [00:07<00:05,  2.40it/s]
2025-09-07T09:12:54.0879466Z running benchmark:  63% 19/30 [00:07<00:04,  2.40it/s]
2025-09-07T09:12:54.5045252Z running benchmark:  67% 20/30 [00:08<00:04,  2.40it/s]
2025-09-07T09:12:54.9218322Z running benchmark:  70% 21/30 [00:08<00:03,  2.40it/s]
2025-09-07T09:12:55.3365730Z running benchmark:  73% 22/30 [00:09<00:03,  2.40it/s]
2025-09-07T09:12:55.7573385Z running benchmark:  77% 23/30 [00:09<00:02,  2.40it/s]
2025-09-07T09:12:56.1756418Z running benchmark:  80% 24/30 [00:10<00:02,  2.40it/s]
2025-09-07T09:12:56.5926076Z running benchmark:  83% 25/30 [00:10<00:02,  2.39it/s]
2025-09-07T09:12:57.0081549Z running benchmark:  87% 26/30 [00:10<00:01,  2.40it/s]
2025-09-07T09:12:57.4231871Z running benchmark:  90% 27/30 [00:11<00:01,  2.40it/s]
2025-09-07T09:12:57.8387326Z running benchmark:  93% 28/30 [00:11<00:00,  2.40it/s]
2025-09-07T09:12:58.2547624Z running benchmark:  97% 29/30 [00:12<00:00,  2.40it/s]
2025-09-07T09:12:58.2549122Z running benchmark: 100% 30/30 [00:12<00:00,  2.40it/s]
2025-09-07T09:12:58.2549488Z running benchmark: 100% 30/30 [00:12<00:00,  2.40it/s]
2025-09-07T09:12:58.3121406Z 1.149x
2025-09-07T09:12:58.3122108Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:13:00.5668425Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:13:00.5669286Z   import pynvml  # type: ignore[import]
2025-09-07T09:13:02.5890790Z 
2025-09-07T09:13:08.4218916Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:13:08.4219488Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:13:08.4486491Z cpu  eval  hf_Roberta_base                    
2025-09-07T09:13:14.5879556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5880009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5880357Z     return mod(**inputs)
2025-09-07T09:13:14.5880787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5881220Z     outputs = self.roberta(
2025-09-07T09:13:14.5881651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T09:13:14.5882084Z     embedding_output = self.embeddings(
2025-09-07T09:13:14.5882502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 90, in forward
2025-09-07T09:13:14.5883383Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
2025-09-07T09:13:14.5884026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1576, in create_position_ids_from_input_ids
2025-09-07T09:13:14.5884519Z     mask = input_ids.ne(padding_idx).int()
2025-09-07T09:13:14.5884657Z 
2025-09-07T09:13:14.5884792Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5885000Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5885191Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5885379Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5885659Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5885857Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5886049Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5886254Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5886456Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5886654Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5886842Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5887040Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5887265Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5887632Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5887956Z     return mod(**inputs)
2025-09-07T09:13:14.5888358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5888773Z     outputs = self.roberta(
2025-09-07T09:13:14.5889182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T09:13:14.5889622Z     embedding_output = self.embeddings(
2025-09-07T09:13:14.5890032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 90, in forward
2025-09-07T09:13:14.5890594Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
2025-09-07T09:13:14.5891207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1577, in create_position_ids_from_input_ids
2025-09-07T09:13:14.5891807Z     incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask
2025-09-07T09:13:14.5892055Z 
2025-09-07T09:13:14.5892168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5892527Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5892852Z     return mod(**inputs)
2025-09-07T09:13:14.5893242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5893661Z     outputs = self.roberta(
2025-09-07T09:13:14.5894155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 817, in forward
2025-09-07T09:13:14.5894647Z     embedding_output = self.embeddings(
2025-09-07T09:13:14.5895058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 90, in forward
2025-09-07T09:13:14.5895623Z     position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length)
2025-09-07T09:13:14.5896234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1577, in create_position_ids_from_input_ids
2025-09-07T09:13:14.5896820Z     incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask
2025-09-07T09:13:14.5897062Z 
2025-09-07T09:13:14.5897164Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5897610Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5897940Z     return mod(**inputs)
2025-09-07T09:13:14.5898331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5898745Z     outputs = self.roberta(
2025-09-07T09:13:14.5899126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5899537Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5899973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5900390Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5900738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5901098Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5901513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5901930Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5902325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5902704Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5903239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5903652Z     self_outputs = self.self(
2025-09-07T09:13:14.5904023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5904399Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5904783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.5905352Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.5905626Z 
2025-09-07T09:13:14.5905727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5906090Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5906410Z     return mod(**inputs)
2025-09-07T09:13:14.5906809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5907240Z     outputs = self.roberta(
2025-09-07T09:13:14.5907632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5908041Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5908446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5908964Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5909306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5909665Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5910076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5910504Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5910877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5911252Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5911636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5912041Z     self_outputs = self.self(
2025-09-07T09:13:14.5912467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5912859Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5913255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.5913661Z     self.key(current_states)
2025-09-07T09:13:14.5913778Z 
2025-09-07T09:13:14.5913885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5914238Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5914623Z     return mod(**inputs)
2025-09-07T09:13:14.5915009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5915414Z     outputs = self.roberta(
2025-09-07T09:13:14.5915795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5916208Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5916608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5917015Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5917350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5917711Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5918116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5918526Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5918918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5919301Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5919701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5920118Z     self_outputs = self.self(
2025-09-07T09:13:14.5920477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5920857Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5921246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.5921653Z     self.value(current_states)
2025-09-07T09:13:14.5921773Z 
2025-09-07T09:13:14.5921850Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5922089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5922456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5922840Z     return mod(**inputs)
2025-09-07T09:13:14.5923236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5923686Z     outputs = self.roberta(
2025-09-07T09:13:14.5924079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5924492Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5924889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5925299Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5925646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5926010Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5926538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5926957Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5927336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5927707Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5928097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5928495Z     self_outputs = self.self(
2025-09-07T09:13:14.5928888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5929266Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5929653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.5930117Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.5930299Z 
2025-09-07T09:13:14.5930403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5930758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5931079Z     return mod(**inputs)
2025-09-07T09:13:14.5931469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5931878Z     outputs = self.roberta(
2025-09-07T09:13:14.5932257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5932669Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5933064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5933478Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5933814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5934178Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5934582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5934999Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5935369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5935732Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5936117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.5936577Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.5937034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.5938582Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.5938720Z 
2025-09-07T09:13:14.5938823Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5939180Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5939501Z     return mod(**inputs)
2025-09-07T09:13:14.5939889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5940300Z     outputs = self.roberta(
2025-09-07T09:13:14.5940684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5941099Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5941542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5941962Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5942299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5942660Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5943067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.5943485Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.5943930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.5944331Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.5944773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.5945263Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.5945726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.5946146Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.5946282Z 
2025-09-07T09:13:14.5946383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5946742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5947064Z     return mod(**inputs)
2025-09-07T09:13:14.5947452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5947858Z     outputs = self.roberta(
2025-09-07T09:13:14.5948238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5948645Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5949049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5949455Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5949791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5950151Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5950557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.5950975Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.5951385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.5951775Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.5952217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.5952782Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.5953234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.5953679Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.5954056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.5954396Z     return self.act(input)
2025-09-07T09:13:14.5954509Z 
2025-09-07T09:13:14.5954608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5954962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5955285Z     return mod(**inputs)
2025-09-07T09:13:14.5955700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5956115Z     outputs = self.roberta(
2025-09-07T09:13:14.5956500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5956908Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5957298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5957702Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5958073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5958440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5958849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.5959255Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.5959658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.5960053Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.5960487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.5960986Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.5961450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.5961869Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.5962013Z 
2025-09-07T09:13:14.5962111Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5962467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5962785Z     return mod(**inputs)
2025-09-07T09:13:14.5963171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5963580Z     outputs = self.roberta(
2025-09-07T09:13:14.5963963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5964371Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5964765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5965171Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5965516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5965875Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5966280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5966763Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5967143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5967517Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5967907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5968313Z     self_outputs = self.self(
2025-09-07T09:13:14.5968662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5969028Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5969415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.5969998Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.5970269Z 
2025-09-07T09:13:14.5970371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5970732Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5971049Z     return mod(**inputs)
2025-09-07T09:13:14.5971436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5971839Z     outputs = self.roberta(
2025-09-07T09:13:14.5972249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5972661Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5973060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5973471Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5973811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5974163Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5974568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5974982Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5975362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5975729Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5976115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5976517Z     self_outputs = self.self(
2025-09-07T09:13:14.5976881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5977256Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5977707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.5978116Z     self.key(current_states)
2025-09-07T09:13:14.5978234Z 
2025-09-07T09:13:14.5978335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5978691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5979009Z     return mod(**inputs)
2025-09-07T09:13:14.5979389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5979801Z     outputs = self.roberta(
2025-09-07T09:13:14.5980184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5980644Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5981067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5981474Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5981815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5982174Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5982578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5982986Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5983360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5983731Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5984156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5984566Z     self_outputs = self.self(
2025-09-07T09:13:14.5984913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5985285Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5985672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.5986073Z     self.value(current_states)
2025-09-07T09:13:14.5986184Z 
2025-09-07T09:13:14.5986308Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.5986544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5986907Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5987237Z     return mod(**inputs)
2025-09-07T09:13:14.5987634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5988042Z     outputs = self.roberta(
2025-09-07T09:13:14.5988430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5988834Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5989230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5989631Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5989972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5990331Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5990744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5991163Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5991535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5991903Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5992291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.5992693Z     self_outputs = self.self(
2025-09-07T09:13:14.5993043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.5993403Z     return func(*args, **kwargs)
2025-09-07T09:13:14.5993788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.5994250Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.5994434Z 
2025-09-07T09:13:14.5994765Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.5995157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.5995473Z     return mod(**inputs)
2025-09-07T09:13:14.5995855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.5996261Z     outputs = self.roberta(
2025-09-07T09:13:14.5996640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.5997082Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.5997480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.5997883Z     layer_outputs = layer_module(
2025-09-07T09:13:14.5998256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.5998622Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.5999022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.5999440Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.5999812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6000180Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6000603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6001066Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6001521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6001942Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6002076Z 
2025-09-07T09:13:14.6002184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6002537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6002850Z     return mod(**inputs)
2025-09-07T09:13:14.6003394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6003823Z     outputs = self.roberta(
2025-09-07T09:13:14.6004226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6004638Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6005045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6005458Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6005812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6006176Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6006577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6006997Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6007403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6007803Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6008241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6008727Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6009183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6009718Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6009850Z 
2025-09-07T09:13:14.6009958Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6010319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6010639Z     return mod(**inputs)
2025-09-07T09:13:14.6011029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6011440Z     outputs = self.roberta(
2025-09-07T09:13:14.6011827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6012224Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6012665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6013078Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6013422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6013780Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6014174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6014589Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6015034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6015439Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6015874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6016357Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6016806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6017245Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6017675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6018012Z     return self.act(input)
2025-09-07T09:13:14.6018124Z 
2025-09-07T09:13:14.6018222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6018578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6018895Z     return mod(**inputs)
2025-09-07T09:13:14.6019276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6019680Z     outputs = self.roberta(
2025-09-07T09:13:14.6020069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6020489Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6020892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6021297Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6021635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6021992Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6022395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6022806Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6023198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6023737Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6024171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6024665Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6025127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6025538Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6025676Z 
2025-09-07T09:13:14.6025775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6026124Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6026443Z     return mod(**inputs)
2025-09-07T09:13:14.6026859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6027265Z     outputs = self.roberta(
2025-09-07T09:13:14.6027648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6028058Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6028450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6028855Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6029215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6029573Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6029969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6030391Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6030765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6031137Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6031533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6031938Z     self_outputs = self.self(
2025-09-07T09:13:14.6032292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6032658Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6033052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6033602Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6033868Z 
2025-09-07T09:13:14.6033977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6034337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6034652Z     return mod(**inputs)
2025-09-07T09:13:14.6035036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6035446Z     outputs = self.roberta(
2025-09-07T09:13:14.6035830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6036241Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6036629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6037034Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6037377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6037814Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6038216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6038638Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6039015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6039391Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6039782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6040179Z     self_outputs = self.self(
2025-09-07T09:13:14.6040529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6040899Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6041337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6041755Z     self.key(current_states)
2025-09-07T09:13:14.6041865Z 
2025-09-07T09:13:14.6041964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6042319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6042638Z     return mod(**inputs)
2025-09-07T09:13:14.6043055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6043461Z     outputs = self.roberta(
2025-09-07T09:13:14.6043846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6044260Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6044670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6045076Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6045410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6045765Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6046170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6046581Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6046953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6047317Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6056702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6057164Z     self_outputs = self.self(
2025-09-07T09:13:14.6057646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6058036Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6058473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6058912Z     self.value(current_states)
2025-09-07T09:13:14.6059030Z 
2025-09-07T09:13:14.6059115Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6059345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6059720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6060049Z     return mod(**inputs)
2025-09-07T09:13:14.6060446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6060987Z     outputs = self.roberta(
2025-09-07T09:13:14.6061422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6061840Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6062243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6062650Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6062989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6063359Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6063770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6064187Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6064607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6064983Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6065381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6065788Z     self_outputs = self.self(
2025-09-07T09:13:14.6066143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6066512Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6066932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6067401Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6067594Z 
2025-09-07T09:13:14.6067698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6068059Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6068371Z     return mod(**inputs)
2025-09-07T09:13:14.6068753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6069162Z     outputs = self.roberta(
2025-09-07T09:13:14.6069548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6069952Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6070360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6070781Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6071128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6071490Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6071903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6072313Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6072690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6073063Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6073454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6073913Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6074372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6074792Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6074974Z 
2025-09-07T09:13:14.6075085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6075476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6075791Z     return mod(**inputs)
2025-09-07T09:13:14.6076173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6076582Z     outputs = self.roberta(
2025-09-07T09:13:14.6076964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6077376Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6077765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6078169Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6078544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6078914Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6079310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6079726Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6080139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6080555Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6081028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6081515Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6081976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6082395Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6082531Z 
2025-09-07T09:13:14.6082638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6082993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6083307Z     return mod(**inputs)
2025-09-07T09:13:14.6083693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6084101Z     outputs = self.roberta(
2025-09-07T09:13:14.6084488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6084895Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6085288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6085696Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6086041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6086401Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6086802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6087215Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6087614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6090365Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6090820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6092707Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6093182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6093646Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6094030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6094396Z     return self.act(input)
2025-09-07T09:13:14.6094516Z 
2025-09-07T09:13:14.6094626Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6095001Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6095335Z     return mod(**inputs)
2025-09-07T09:13:14.6095763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6096188Z     outputs = self.roberta(
2025-09-07T09:13:14.6096638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6097052Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6097456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6097958Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6098325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6098699Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6099152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6099585Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6099999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6100404Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6100861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6101360Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6101828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6102249Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6102388Z 
2025-09-07T09:13:14.6102495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6102860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6103342Z     return mod(**inputs)
2025-09-07T09:13:14.6103735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6104174Z     outputs = self.roberta(
2025-09-07T09:13:14.6104552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6104960Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6105360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6105766Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6106101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6106464Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6106987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6107402Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6107849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6108223Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6108615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6109020Z     self_outputs = self.self(
2025-09-07T09:13:14.6109377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6109745Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6110131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6110681Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6110952Z 
2025-09-07T09:13:14.6111053Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6111476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6111805Z     return mod(**inputs)
2025-09-07T09:13:14.6112185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6112592Z     outputs = self.roberta(
2025-09-07T09:13:14.6112982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6113391Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6113833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6114254Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6114600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6114968Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6115385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6115808Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6116187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6116558Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6116950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6117357Z     self_outputs = self.self(
2025-09-07T09:13:14.6117704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6118071Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6118463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6118864Z     self.key(current_states)
2025-09-07T09:13:14.6118973Z 
2025-09-07T09:13:14.6119074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6119432Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6119754Z     return mod(**inputs)
2025-09-07T09:13:14.6120145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6120558Z     outputs = self.roberta(
2025-09-07T09:13:14.6120994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6121407Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6121805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6122269Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6122610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6122964Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6123370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6123782Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6124155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6124521Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6124909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6125317Z     self_outputs = self.self(
2025-09-07T09:13:14.6125710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6126090Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6126476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6126882Z     self.value(current_states)
2025-09-07T09:13:14.6127001Z 
2025-09-07T09:13:14.6127078Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6127311Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6127694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6128022Z     return mod(**inputs)
2025-09-07T09:13:14.6128402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6128809Z     outputs = self.roberta(
2025-09-07T09:13:14.6129191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6129599Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6129994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6130398Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6130743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6131103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6131513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6131919Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6132283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6132647Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6133027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6133424Z     self_outputs = self.self(
2025-09-07T09:13:14.6133762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6134121Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6134493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6135236Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6135420Z 
2025-09-07T09:13:14.6135522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6135870Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6136240Z     return mod(**inputs)
2025-09-07T09:13:14.6136620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6137025Z     outputs = self.roberta(
2025-09-07T09:13:14.6137393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6137867Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6138257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6138658Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6138995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6139345Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6139787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6140196Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6140565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6140931Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6141306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6141762Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6142250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6142678Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6142818Z 
2025-09-07T09:13:14.6142920Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6143268Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6143589Z     return mod(**inputs)
2025-09-07T09:13:14.6143974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6144380Z     outputs = self.roberta(
2025-09-07T09:13:14.6144759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6145163Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6145557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6145956Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6146290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6146641Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6147046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6147457Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6147850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6148236Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6148665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6149193Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6149643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6150093Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6150225Z 
2025-09-07T09:13:14.6150327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6150671Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6150986Z     return mod(**inputs)
2025-09-07T09:13:14.6151363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6151761Z     outputs = self.roberta(
2025-09-07T09:13:14.6152132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6152533Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6152917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6153321Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6153690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6154044Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6154448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6154858Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6155249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6155638Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6156094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6156576Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6157021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6157461Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6157831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6158157Z     return self.act(input)
2025-09-07T09:13:14.6158260Z 
2025-09-07T09:13:14.6158356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6158697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6159010Z     return mod(**inputs)
2025-09-07T09:13:14.6159384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6159789Z     outputs = self.roberta(
2025-09-07T09:13:14.6160161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6160561Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6160946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6161339Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6161667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6162014Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6162409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6162865Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6163254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6163641Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6164125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6164614Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6165068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6165477Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6165608Z 
2025-09-07T09:13:14.6165705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6166050Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6166363Z     return mod(**inputs)
2025-09-07T09:13:14.6166733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6167136Z     outputs = self.roberta(
2025-09-07T09:13:14.6167556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6167963Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6168351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6168745Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6169077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6169427Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6169859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6170269Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6170634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6171001Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6171385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6171782Z     self_outputs = self.self(
2025-09-07T09:13:14.6172126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6172486Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6172860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6173400Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6173664Z 
2025-09-07T09:13:14.6173763Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6174101Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6174414Z     return mod(**inputs)
2025-09-07T09:13:14.6174786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6175187Z     outputs = self.roberta(
2025-09-07T09:13:14.6175561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6175959Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6176346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6176793Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6177124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6177469Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6177954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6178361Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6178730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6179093Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6179472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6179867Z     self_outputs = self.self(
2025-09-07T09:13:14.6180213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6180570Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6180947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6181378Z     self.key(current_states)
2025-09-07T09:13:14.6181489Z 
2025-09-07T09:13:14.6181583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6181926Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6182236Z     return mod(**inputs)
2025-09-07T09:13:14.6182603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6182999Z     outputs = self.roberta(
2025-09-07T09:13:14.6183406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6183812Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6184204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6184607Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6184958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6185318Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6185720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6186126Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6186490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6186852Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6187238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6187640Z     self_outputs = self.self(
2025-09-07T09:13:14.6187980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6188342Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6188720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6189122Z     self.value(current_states)
2025-09-07T09:13:14.6189231Z 
2025-09-07T09:13:14.6189311Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6189531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6189872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6190185Z     return mod(**inputs)
2025-09-07T09:13:14.6190604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6191002Z     outputs = self.roberta(
2025-09-07T09:13:14.6191378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6191823Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6192213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6192614Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6192939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6193290Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6193689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6194100Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6194460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6194817Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6195227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6195625Z     self_outputs = self.self(
2025-09-07T09:13:14.6195969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6196323Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6196697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6197154Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6197371Z 
2025-09-07T09:13:14.6197465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6197802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6198108Z     return mod(**inputs)
2025-09-07T09:13:14.6198482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6198880Z     outputs = self.roberta(
2025-09-07T09:13:14.6199248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6199645Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6200025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6200412Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6200735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6201089Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6201495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6201915Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6202281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6202673Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6203202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6203663Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6204122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6204635Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6204773Z 
2025-09-07T09:13:14.6204877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6205225Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6205614Z     return mod(**inputs)
2025-09-07T09:13:14.6205993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6206395Z     outputs = self.roberta(
2025-09-07T09:13:14.6206770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6207175Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6207560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6207984Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6208323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6208680Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6209126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6209560Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6209975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6210375Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6210819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6211301Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6211804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6212239Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6212384Z 
2025-09-07T09:13:14.6212491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6212853Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6213177Z     return mod(**inputs)
2025-09-07T09:13:14.6213560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6213963Z     outputs = self.roberta(
2025-09-07T09:13:14.6214348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6214759Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6215147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6215556Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6215894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6216252Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6216647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6217063Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6217458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6217919Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6218360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6218897Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6219344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6219827Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6220208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6220543Z     return self.act(input)
2025-09-07T09:13:14.6220645Z 
2025-09-07T09:13:14.6220741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6221092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6221412Z     return mod(**inputs)
2025-09-07T09:13:14.6221795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6222206Z     outputs = self.roberta(
2025-09-07T09:13:14.6222582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6222984Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6223418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6223823Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6224158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6224510Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6224911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6225324Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6225750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6226145Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6226580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6227073Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6227534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6227944Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6228073Z 
2025-09-07T09:13:14.6228168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6228519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6228832Z     return mod(**inputs)
2025-09-07T09:13:14.6229211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6229614Z     outputs = self.roberta(
2025-09-07T09:13:14.6229990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6230397Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6230791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6231196Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6231520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6231873Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6232273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6232731Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6233105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6233512Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6233899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6234300Z     self_outputs = self.self(
2025-09-07T09:13:14.6234653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6235020Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6235396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6235941Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6236210Z 
2025-09-07T09:13:14.6236307Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6236664Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6236981Z     return mod(**inputs)
2025-09-07T09:13:14.6237403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6237816Z     outputs = self.roberta(
2025-09-07T09:13:14.6238203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6238610Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6239001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6239401Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6239778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6240166Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6240567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6240978Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6241352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6241721Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6242105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6242500Z     self_outputs = self.self(
2025-09-07T09:13:14.6242847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6243217Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6243597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6243999Z     self.key(current_states)
2025-09-07T09:13:14.6244107Z 
2025-09-07T09:13:14.6244209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6244556Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6244874Z     return mod(**inputs)
2025-09-07T09:13:14.6245247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6245651Z     outputs = self.roberta(
2025-09-07T09:13:14.6246025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6246430Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6246866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6247269Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6247635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6247987Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6248389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6248798Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6249162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6249526Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6249907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6250311Z     self_outputs = self.self(
2025-09-07T09:13:14.6250657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6251022Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6251433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6251846Z     self.value(current_states)
2025-09-07T09:13:14.6251958Z 
2025-09-07T09:13:14.6252035Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6252263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6252607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6252924Z     return mod(**inputs)
2025-09-07T09:13:14.6253329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6253739Z     outputs = self.roberta(
2025-09-07T09:13:14.6254109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6254509Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6254904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6255304Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6255639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6255987Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6256387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6256800Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6257168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6257621Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6258007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6258406Z     self_outputs = self.self(
2025-09-07T09:13:14.6258754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6259120Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6259498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6259949Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6260133Z 
2025-09-07T09:13:14.6260231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6260637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6260957Z     return mod(**inputs)
2025-09-07T09:13:14.6261347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6261794Z     outputs = self.roberta(
2025-09-07T09:13:14.6262173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6262579Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6262967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6263362Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6263701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6264057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6264455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6264865Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6265266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6265641Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6266023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6266479Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6266934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6267345Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6267535Z 
2025-09-07T09:13:14.6267633Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6267983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6268308Z     return mod(**inputs)
2025-09-07T09:13:14.6268686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6269092Z     outputs = self.roberta(
2025-09-07T09:13:14.6269468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6269870Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6270261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6270660Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6270999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6271357Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6271774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6272192Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6272585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6272981Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6273419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6273904Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6274359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6274820Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6274962Z 
2025-09-07T09:13:14.6275058Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6275445Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6275762Z     return mod(**inputs)
2025-09-07T09:13:14.6276142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6276548Z     outputs = self.roberta(
2025-09-07T09:13:14.6276929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6277334Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6277734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6278135Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6278471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6278827Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6279277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6279697Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6280092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6280487Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6280916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6281467Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6281919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6282353Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6282738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6283079Z     return self.act(input)
2025-09-07T09:13:14.6283186Z 
2025-09-07T09:13:14.6283289Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6283633Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6283953Z     return mod(**inputs)
2025-09-07T09:13:14.6284334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6284741Z     outputs = self.roberta(
2025-09-07T09:13:14.6285119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6285521Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6285918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6286320Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6286657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6287012Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6287407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6287821Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6288217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6288660Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6289082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6289627Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6290093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6290503Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6290634Z 
2025-09-07T09:13:14.6290735Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6291074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6291143Z     return mod(**inputs)
2025-09-07T09:13:14.6291424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6291499Z     outputs = self.roberta(
2025-09-07T09:13:14.6291774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6291850Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6292169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6292243Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6292473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6292550Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6292832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6292905Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6293186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6293267Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6293551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6293628Z     self_outputs = self.self(
2025-09-07T09:13:14.6293870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6293937Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6294217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6294426Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6294430Z 
2025-09-07T09:13:14.6294534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6294726Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6294792Z     return mod(**inputs)
2025-09-07T09:13:14.6295072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6295137Z     outputs = self.roberta(
2025-09-07T09:13:14.6295416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6295484Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6295761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6295827Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6296047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6296165Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6296439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6296564Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6296805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6296875Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6297146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6297211Z     self_outputs = self.self(
2025-09-07T09:13:14.6297454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6297516Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6297843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6297912Z     self.key(current_states)
2025-09-07T09:13:14.6297915Z 
2025-09-07T09:13:14.6298013Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6298246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6298308Z     return mod(**inputs)
2025-09-07T09:13:14.6298594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6298658Z     outputs = self.roberta(
2025-09-07T09:13:14.6298938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6299006Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6299480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6299557Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6299775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6299852Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6300124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6300197Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6300448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6300514Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6300798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6300866Z     self_outputs = self.self(
2025-09-07T09:13:14.6301112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6301180Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6301458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6301531Z     self.value(current_states)
2025-09-07T09:13:14.6301535Z 
2025-09-07T09:13:14.6301612Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6301718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6301913Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6301974Z     return mod(**inputs)
2025-09-07T09:13:14.6302260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6302324Z     outputs = self.roberta(
2025-09-07T09:13:14.6302655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6302726Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6303168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6303244Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6303459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6303536Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6303808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6303883Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6304132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6304199Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6304479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6304544Z     self_outputs = self.self(
2025-09-07T09:13:14.6304852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6304915Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6305188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6305324Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6305328Z 
2025-09-07T09:13:14.6305421Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6305661Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6305722Z     return mod(**inputs)
2025-09-07T09:13:14.6306003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6306076Z     outputs = self.roberta(
2025-09-07T09:13:14.6306353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6306428Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6306699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6306771Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6306993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6307063Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6307344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6307416Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6307661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6307725Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6307997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6308126Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6308398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6308480Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6308484Z 
2025-09-07T09:13:14.6308577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6308840Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6308907Z     return mod(**inputs)
2025-09-07T09:13:14.6309201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6309324Z     outputs = self.roberta(
2025-09-07T09:13:14.6309597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6309668Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6309944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6310007Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6310230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6310305Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6310582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6310661Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6310966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6311037Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6311348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6311470Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6311745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6311861Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6311866Z 
2025-09-07T09:13:14.6311961Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6312152Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6312219Z     return mod(**inputs)
2025-09-07T09:13:14.6312497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6312567Z     outputs = self.roberta(
2025-09-07T09:13:14.6312837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6312909Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6313182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6313244Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6313468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6313537Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6313815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6313892Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6314153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6314232Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6314537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6314656Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6314931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6315084Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6315300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6315404Z     return self.act(input)
2025-09-07T09:13:14.6315408Z 
2025-09-07T09:13:14.6315511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6315698Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6315763Z     return mod(**inputs)
2025-09-07T09:13:14.6316038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6316102Z     outputs = self.roberta(
2025-09-07T09:13:14.6316383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6316451Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6316727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6316795Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6317048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6317127Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6317400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6317480Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6317737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6317811Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6318150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6318282Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6318566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6318638Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6318642Z 
2025-09-07T09:13:14.6318741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6318927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6318994Z     return mod(**inputs)
2025-09-07T09:13:14.6319270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6319329Z     outputs = self.roberta(
2025-09-07T09:13:14.6319620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6319689Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6319965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6320033Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6320247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6320324Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6320594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6320673Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6320918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6321028Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6321315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6321417Z     self_outputs = self.self(
2025-09-07T09:13:14.6321667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6321729Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6322009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6322219Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6322223Z 
2025-09-07T09:13:14.6322320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6322517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6322577Z     return mod(**inputs)
2025-09-07T09:13:14.6322857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6322921Z     outputs = self.roberta(
2025-09-07T09:13:14.6323229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6323302Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6323576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6323648Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6323860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6323935Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6324242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6324316Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6324574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6324637Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6324916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6324980Z     self_outputs = self.self(
2025-09-07T09:13:14.6325217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6325285Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6325554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6325627Z     self.key(current_states)
2025-09-07T09:13:14.6325630Z 
2025-09-07T09:13:14.6325719Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6325911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6325972Z     return mod(**inputs)
2025-09-07T09:13:14.6326247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6326315Z     outputs = self.roberta(
2025-09-07T09:13:14.6326602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6326675Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6326947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6327046Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6327266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6327335Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6327672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6327745Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6327981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6328049Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6328320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6328388Z     self_outputs = self.self(
2025-09-07T09:13:14.6328623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6328691Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6328960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6329026Z     self.value(current_states)
2025-09-07T09:13:14.6329063Z 
2025-09-07T09:13:14.6329143Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6329232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6329418Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6329474Z     return mod(**inputs)
2025-09-07T09:13:14.6329748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6329819Z     outputs = self.roberta(
2025-09-07T09:13:14.6330120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6330190Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6330461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6330531Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6330755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6330825Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6331097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6331168Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6331415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6331476Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6331746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6331824Z     self_outputs = self.self(
2025-09-07T09:13:14.6332061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6332124Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6332411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6332541Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6332546Z 
2025-09-07T09:13:14.6332637Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6332819Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6332881Z     return mod(**inputs)
2025-09-07T09:13:14.6333212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6333279Z     outputs = self.roberta(
2025-09-07T09:13:14.6333599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6333668Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6333940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6334003Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6334218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6334286Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6334559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6334632Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6334891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6334976Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6335282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6335420Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6340680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6340768Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6340772Z 
2025-09-07T09:13:14.6340881Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6341165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6341233Z     return mod(**inputs)
2025-09-07T09:13:14.6341536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6341603Z     outputs = self.roberta(
2025-09-07T09:13:14.6341884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6341950Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6342225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6342289Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6342515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6342594Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6342875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6342955Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6343221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6343293Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6343604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6343719Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6343997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6344073Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6344077Z 
2025-09-07T09:13:14.6344179Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6344429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6344489Z     return mod(**inputs)
2025-09-07T09:13:14.6344805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6344872Z     outputs = self.roberta(
2025-09-07T09:13:14.6345150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6345216Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6345495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6345562Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6345787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6345872Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6346148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6346232Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6346531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6346605Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6346919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6347033Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6347312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6347454Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6347683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6347753Z     return self.act(input)
2025-09-07T09:13:14.6347759Z 
2025-09-07T09:13:14.6347863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6348074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6348131Z     return mod(**inputs)
2025-09-07T09:13:14.6348431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6348495Z     outputs = self.roberta(
2025-09-07T09:13:14.6348769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6348840Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6349113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6349185Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6349405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6349484Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6349757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6349835Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6350104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6350173Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6350488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6350653Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6350929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6351045Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6351048Z 
2025-09-07T09:13:14.6351146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6351349Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6351410Z     return mod(**inputs)
2025-09-07T09:13:14.6351695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6351756Z     outputs = self.roberta(
2025-09-07T09:13:14.6352031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6352103Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6352371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6352442Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6352696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6352770Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6353049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6353123Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6353374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6353472Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6353746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6353819Z     self_outputs = self.self(
2025-09-07T09:13:14.6354060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6354125Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6354393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6354613Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6354617Z 
2025-09-07T09:13:14.6354715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6354916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6354979Z     return mod(**inputs)
2025-09-07T09:13:14.6355276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6355343Z     outputs = self.roberta(
2025-09-07T09:13:14.6355618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6355682Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6355955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6356018Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6356241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6356310Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6356586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6356714Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6356954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6357052Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6357329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6357396Z     self_outputs = self.self(
2025-09-07T09:13:14.6357631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6357689Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6357962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6358024Z     self.key(current_states)
2025-09-07T09:13:14.6358031Z 
2025-09-07T09:13:14.6358127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6358317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6358382Z     return mod(**inputs)
2025-09-07T09:13:14.6358690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6358750Z     outputs = self.roberta(
2025-09-07T09:13:14.6359025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6359093Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6359367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6359427Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6359675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6359746Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6360017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6360095Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6360331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6360391Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6360667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6360730Z     self_outputs = self.self(
2025-09-07T09:13:14.6360967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6361030Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6361305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6361367Z     self.value(current_states)
2025-09-07T09:13:14.6361372Z 
2025-09-07T09:13:14.6361450Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6361550Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6361733Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6361796Z     return mod(**inputs)
2025-09-07T09:13:14.6362068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6362130Z     outputs = self.roberta(
2025-09-07T09:13:14.6362406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6362504Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6362783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6362876Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6363096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6363165Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6363431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6363509Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6363744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6363809Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6364082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6364145Z     self_outputs = self.self(
2025-09-07T09:13:14.6364380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6364438Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6364747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6364874Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6364877Z 
2025-09-07T09:13:14.6364973Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6365159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6365216Z     return mod(**inputs)
2025-09-07T09:13:14.6365526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6365587Z     outputs = self.roberta(
2025-09-07T09:13:14.6365862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6365929Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6366200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6366266Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6366477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6366551Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6366819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6366892Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6367135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6367195Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6367470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6367588Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6367860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6367936Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6367939Z 
2025-09-07T09:13:14.6368027Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6368217Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6368274Z     return mod(**inputs)
2025-09-07T09:13:14.6368590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6368648Z     outputs = self.roberta(
2025-09-07T09:13:14.6368955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6369023Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6369293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6369357Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6369573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6369643Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6369915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6369989Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6370252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6370324Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6370670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6370788Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6371061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6371137Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6371141Z 
2025-09-07T09:13:14.6371229Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6371458Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6371518Z     return mod(**inputs)
2025-09-07T09:13:14.6371801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6371862Z     outputs = self.roberta(
2025-09-07T09:13:14.6372133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6372200Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6372467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6372532Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6372745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6372816Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6373093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6373165Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6373431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6373498Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6373804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6373910Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6374177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6374286Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6374532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6374597Z     return self.act(input)
2025-09-07T09:13:14.6374600Z 
2025-09-07T09:13:14.6374690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6374911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6374969Z     return mod(**inputs)
2025-09-07T09:13:14.6375246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6375310Z     outputs = self.roberta(
2025-09-07T09:13:14.6375582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6375648Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6375920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6375985Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6376195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6376262Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6376570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6376641Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6376899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6376962Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6377264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6377436Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6377776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6377856Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6377860Z 
2025-09-07T09:13:14.6377951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6378144Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6378201Z     return mod(**inputs)
2025-09-07T09:13:14.6378475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6378535Z     outputs = self.roberta(
2025-09-07T09:13:14.6378805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6378873Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6379143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6379206Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6379424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6379495Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6379771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6379841Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6380096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6380157Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6380426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6380528Z     self_outputs = self.self(
2025-09-07T09:13:14.6380762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6380858Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6381131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6381339Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6381343Z 
2025-09-07T09:13:14.6381431Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6381617Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6381674Z     return mod(**inputs)
2025-09-07T09:13:14.6381948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6382012Z     outputs = self.roberta(
2025-09-07T09:13:14.6382279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6382474Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6382750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6382810Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6383025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6383088Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6383358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6383463Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6383699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6383762Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6384037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6384099Z     self_outputs = self.self(
2025-09-07T09:13:14.6384333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6384392Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6384662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6384723Z     self.key(current_states)
2025-09-07T09:13:14.6384726Z 
2025-09-07T09:13:14.6384820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6385004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6385060Z     return mod(**inputs)
2025-09-07T09:13:14.6385335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6385397Z     outputs = self.roberta(
2025-09-07T09:13:14.6385667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6385728Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6385998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6386058Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6386267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6386377Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6386647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6386753Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6386991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6387049Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6387320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6387379Z     self_outputs = self.self(
2025-09-07T09:13:14.6387614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6387671Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6387941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6388006Z     self.value(current_states)
2025-09-07T09:13:14.6388009Z 
2025-09-07T09:13:14.6388081Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6388172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6388421Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6388480Z     return mod(**inputs)
2025-09-07T09:13:14.6388753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6388812Z     outputs = self.roberta(
2025-09-07T09:13:14.6389082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6389144Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6389456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6389519Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6389731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6389802Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6390071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6390141Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6390373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6390433Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6390700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6390761Z     self_outputs = self.self(
2025-09-07T09:13:14.6390993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6391051Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6391324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6391445Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6391448Z 
2025-09-07T09:13:14.6391535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6391718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6391772Z     return mod(**inputs)
2025-09-07T09:13:14.6392046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6392105Z     outputs = self.roberta(
2025-09-07T09:13:14.6392423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6392493Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6392803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6392865Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6393076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6393141Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6393409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6393476Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6393711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6393769Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6394039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6394193Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6394466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6394540Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6394543Z 
2025-09-07T09:13:14.6394630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6394814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6394869Z     return mod(**inputs)
2025-09-07T09:13:14.6395179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6395240Z     outputs = self.roberta(
2025-09-07T09:13:14.6395510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6395575Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6395864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6395931Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6396144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6396210Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6396485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6396559Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6396822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6396888Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6397198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6397308Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6397576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6397649Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6397652Z 
2025-09-07T09:13:14.6397739Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6397923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6398013Z     return mod(**inputs)
2025-09-07T09:13:14.6398289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6398347Z     outputs = self.roberta(
2025-09-07T09:13:14.6398651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6398715Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6398990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6399051Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6399261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6399328Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6399601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6399675Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6399935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6400050Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6400354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6400459Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6400732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6400833Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6401078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6401142Z     return self.act(input)
2025-09-07T09:13:14.6401146Z 
2025-09-07T09:13:14.6401236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6401424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6401480Z     return mod(**inputs)
2025-09-07T09:13:14.6401760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6401817Z     outputs = self.roberta(
2025-09-07T09:13:14.6402087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6402149Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6402418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6402483Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6402697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6402766Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6403191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6403262Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6403522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6403587Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6403892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6404014Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6404286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6404444Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6404448Z 
2025-09-07T09:13:14.6404579Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6404765Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6404820Z     return mod(**inputs)
2025-09-07T09:13:14.6405097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6405153Z     outputs = self.roberta(
2025-09-07T09:13:14.6405420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6405487Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6405756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6405822Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6406031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6406144Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6406422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6406492Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6406731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6406791Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6407061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6407172Z     self_outputs = self.self(
2025-09-07T09:13:14.6407412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6407477Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6407759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6407968Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6407972Z 
2025-09-07T09:13:14.6408070Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6408262Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6408322Z     return mod(**inputs)
2025-09-07T09:13:14.6408600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6408665Z     outputs = self.roberta(
2025-09-07T09:13:14.6408935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6409004Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6409274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6409338Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6409561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6409632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6409912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6409984Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6410245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6410359Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6410632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6410735Z     self_outputs = self.self(
2025-09-07T09:13:14.6410969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6411031Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6411302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6411363Z     self.key(current_states)
2025-09-07T09:13:14.6411367Z 
2025-09-07T09:13:14.6411462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6411650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6411711Z     return mod(**inputs)
2025-09-07T09:13:14.6411981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6412043Z     outputs = self.roberta(
2025-09-07T09:13:14.6412350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6412417Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6412687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6412748Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6412960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6413027Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6413329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6413403Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6413639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6413705Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6413977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6414037Z     self_outputs = self.self(
2025-09-07T09:13:14.6414272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6414331Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6414602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6414668Z     self.value(current_states)
2025-09-07T09:13:14.6414671Z 
2025-09-07T09:13:14.6414745Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6414838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6415025Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6415086Z     return mod(**inputs)
2025-09-07T09:13:14.6415358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6415416Z     outputs = self.roberta(
2025-09-07T09:13:14.6415687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6415750Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6416023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6416120Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6416335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6416437Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6416710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6416785Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6417022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6417082Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6417351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6417409Z     self_outputs = self.self(
2025-09-07T09:13:14.6417689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6417751Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6418024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6418192Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6418196Z 
2025-09-07T09:13:14.6418294Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6418480Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6418535Z     return mod(**inputs)
2025-09-07T09:13:14.6418810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6418870Z     outputs = self.roberta(
2025-09-07T09:13:14.6419177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6419242Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6419511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6419578Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6419791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6419860Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6420129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6420199Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6420436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6420498Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6420769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6420890Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6421165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6421240Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6421243Z 
2025-09-07T09:13:14.6421333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6421519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6421576Z     return mod(**inputs)
2025-09-07T09:13:14.6421853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6421946Z     outputs = self.roberta(
2025-09-07T09:13:14.6422217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6422282Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6422603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6422670Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6422881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6422949Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6423217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6423290Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6423556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6423625Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6423937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6424081Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6424355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6424430Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6424434Z 
2025-09-07T09:13:14.6424525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6424709Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6424765Z     return mod(**inputs)
2025-09-07T09:13:14.6425075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6425137Z     outputs = self.roberta(
2025-09-07T09:13:14.6425407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6425478Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6425747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6425814Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6426028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6426095Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6426366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6426441Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6426701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6426767Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6427072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6427181Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6427453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6427557Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6427767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6427830Z     return self.act(input)
2025-09-07T09:13:14.6427869Z 
2025-09-07T09:13:14.6427958Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6428141Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6428233Z     return mod(**inputs)
2025-09-07T09:13:14.6428509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6428569Z     outputs = self.roberta(
2025-09-07T09:13:14.6428837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6428902Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6429186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6429247Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6429462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6429528Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6429797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6429901Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6430160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6430226Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6430526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6430653Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6430958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6431033Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6431037Z 
2025-09-07T09:13:14.6431124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6431310Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6431372Z     return mod(**inputs)
2025-09-07T09:13:14.6431643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6431703Z     outputs = self.roberta(
2025-09-07T09:13:14.6431972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6432034Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6432306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6432369Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6432583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6432648Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6432920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6432990Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6433228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6433289Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6433557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6433619Z     self_outputs = self.self(
2025-09-07T09:13:14.6433854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6433948Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6434221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 325, in forward
2025-09-07T09:13:14.6434456Z     self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2)
2025-09-07T09:13:14.6434460Z 
2025-09-07T09:13:14.6434551Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6434733Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6434791Z     return mod(**inputs)
2025-09-07T09:13:14.6435063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6435121Z     outputs = self.roberta(
2025-09-07T09:13:14.6435393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6435457Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6435726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6435825Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6436040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6436107Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6436375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6436447Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6436680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6436775Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6437050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6437112Z     self_outputs = self.self(
2025-09-07T09:13:14.6437350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6437408Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6437682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 353, in forward
2025-09-07T09:13:14.6437743Z     self.key(current_states)
2025-09-07T09:13:14.6437746Z 
2025-09-07T09:13:14.6437835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6438022Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6438076Z     return mod(**inputs)
2025-09-07T09:13:14.6438353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6438410Z     outputs = self.roberta(
2025-09-07T09:13:14.6438682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6438747Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6439016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6439078Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6439287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6439354Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6439624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6439740Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6439974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6440066Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6440339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6440401Z     self_outputs = self.self(
2025-09-07T09:13:14.6440632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6440693Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6440960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 358, in forward
2025-09-07T09:13:14.6441023Z     self.value(current_states)
2025-09-07T09:13:14.6441026Z 
2025-09-07T09:13:14.6441103Z cudagraph partition due to non gpu ops
2025-09-07T09:13:14.6441191Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6441374Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6441432Z     return mod(**inputs)
2025-09-07T09:13:14.6441739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6441798Z     outputs = self.roberta(
2025-09-07T09:13:14.6442069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6442131Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6442399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6442465Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6442707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6442775Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6443043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6443115Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6443353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6443412Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6443681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 467, in forward
2025-09-07T09:13:14.6443741Z     self_outputs = self.self(
2025-09-07T09:13:14.6443976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6444039Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6444309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 389, in forward
2025-09-07T09:13:14.6444434Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:13:14.6444437Z 
2025-09-07T09:13:14.6444526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6444712Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6444767Z     return mod(**inputs)
2025-09-07T09:13:14.6445039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6445101Z     outputs = self.roberta(
2025-09-07T09:13:14.6445371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6445487Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6445757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6445850Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6446070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6446137Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6446408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 540, in forward
2025-09-07T09:13:14.6446477Z     self_attention_outputs = self.attention(
2025-09-07T09:13:14.6446714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
2025-09-07T09:13:14.6446773Z     return func(*args, **kwargs)
2025-09-07T09:13:14.6447040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 477, in forward
2025-09-07T09:13:14.6447160Z     attention_output = self.output(self_outputs[0], hidden_states)
2025-09-07T09:13:14.6447429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 413, in forward
2025-09-07T09:13:14.6447539Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6447542Z 
2025-09-07T09:13:14.6447630Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6447814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6447874Z     return mod(**inputs)
2025-09-07T09:13:14.6448147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6448209Z     outputs = self.roberta(
2025-09-07T09:13:14.6448512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6448582Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6448853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6448918Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6449136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6449203Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6449475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6449549Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6449808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6449880Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6450183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6450298Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6450567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 493, in forward
2025-09-07T09:13:14.6450638Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6450641Z 
2025-09-07T09:13:14.6450730Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6450911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6450969Z     return mod(**inputs)
2025-09-07T09:13:14.6451243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6451338Z     outputs = self.roberta(
2025-09-07T09:13:14.6451610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6451706Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6451981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6452043Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6452257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6452325Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6452597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6452666Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6452922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6452992Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6453292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 578, in feed_forward_chunk
2025-09-07T09:13:14.6453437Z     intermediate_output = self.intermediate(attention_output)
2025-09-07T09:13:14.6453712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 494, in forward
2025-09-07T09:13:14.6453812Z     hidden_states = self.intermediate_act_fn(hidden_states)
2025-09-07T09:13:14.6454023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:13:14.6454083Z     return self.act(input)
2025-09-07T09:13:14.6454086Z 
2025-09-07T09:13:14.6454178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6454397Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6454457Z     return mod(**inputs)
2025-09-07T09:13:14.6454736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1094, in forward
2025-09-07T09:13:14.6454799Z     outputs = self.roberta(
2025-09-07T09:13:14.6455072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 881, in forward
2025-09-07T09:13:14.6455134Z     encoder_outputs = self.encoder(
2025-09-07T09:13:14.6455406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 632, in forward
2025-09-07T09:13:14.6455464Z     layer_outputs = layer_module(
2025-09-07T09:13:14.6455676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:14.6455746Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:14.6456014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 570, in forward
2025-09-07T09:13:14.6456090Z     layer_output = apply_chunking_to_forward(
2025-09-07T09:13:14.6456345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward
2025-09-07T09:13:14.6456412Z     return forward_fn(*input_tensors)
2025-09-07T09:13:14.6456711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 579, in feed_forward_chunk
2025-09-07T09:13:14.6456837Z     layer_output = self.output(intermediate_output, attention_output)
2025-09-07T09:13:14.6457109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 507, in forward
2025-09-07T09:13:14.6457180Z     hidden_states = self.dense(hidden_states)
2025-09-07T09:13:14.6457229Z 
2025-09-07T09:13:14.6457320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6457504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6457781Z     return mod(**inputs)
2025-09-07T09:13:14.6458064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1108, in forward
2025-09-07T09:13:14.6458153Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T09:13:14.6458431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1143, in forward
2025-09-07T09:13:14.6458492Z     x = self.dense(features)
2025-09-07T09:13:14.6458496Z 
2025-09-07T09:13:14.6458588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:14.6458771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:14.6458830Z     return mod(**inputs)
2025-09-07T09:13:14.6459104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1108, in forward
2025-09-07T09:13:14.6459192Z     prediction_scores = self.lm_head(sequence_output)
2025-09-07T09:13:14.6459514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1148, in forward
2025-09-07T09:13:14.6459575Z     x = self.decoder(x)
2025-09-07T09:13:14.6459578Z 
2025-09-07T09:13:22.3546033Z 
2025-09-07T09:13:22.5312850Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:13:22.7043039Z running benchmark:   3% 1/30 [00:00<00:05,  5.66it/s]
2025-09-07T09:13:22.8725722Z running benchmark:   7% 2/30 [00:00<00:04,  5.73it/s]
2025-09-07T09:13:23.0349428Z running benchmark:  10% 3/30 [00:00<00:04,  5.82it/s]
2025-09-07T09:13:23.2033051Z running benchmark:  13% 4/30 [00:00<00:04,  5.95it/s]
2025-09-07T09:13:23.3733919Z running benchmark:  17% 5/30 [00:00<00:04,  5.95it/s]
2025-09-07T09:13:23.5441304Z running benchmark:  20% 6/30 [00:01<00:04,  5.92it/s]
2025-09-07T09:13:23.7045386Z running benchmark:  23% 7/30 [00:01<00:03,  5.90it/s]
2025-09-07T09:13:23.8738391Z running benchmark:  27% 8/30 [00:01<00:03,  6.00it/s]
2025-09-07T09:13:24.0446716Z running benchmark:  30% 9/30 [00:01<00:03,  5.97it/s]
2025-09-07T09:13:24.2132120Z running benchmark:  33% 10/30 [00:01<00:03,  5.94it/s]
2025-09-07T09:13:24.3846331Z running benchmark:  37% 11/30 [00:01<00:03,  5.94it/s]
2025-09-07T09:13:24.5538374Z running benchmark:  40% 12/30 [00:02<00:03,  5.90it/s]
2025-09-07T09:13:24.7220016Z running benchmark:  43% 13/30 [00:02<00:02,  5.91it/s]
2025-09-07T09:13:24.8932006Z running benchmark:  47% 14/30 [00:02<00:02,  5.92it/s]
2025-09-07T09:13:25.0605212Z running benchmark:  50% 15/30 [00:02<00:02,  5.89it/s]
2025-09-07T09:13:25.2322250Z running benchmark:  53% 16/30 [00:02<00:02,  5.92it/s]
2025-09-07T09:13:25.4030173Z running benchmark:  57% 17/30 [00:02<00:02,  5.89it/s]
2025-09-07T09:13:25.5775816Z running benchmark:  60% 18/30 [00:03<00:02,  5.88it/s]
2025-09-07T09:13:25.7532819Z running benchmark:  63% 19/30 [00:03<00:01,  5.83it/s]
2025-09-07T09:13:25.9234705Z running benchmark:  67% 20/30 [00:03<00:01,  5.79it/s]
2025-09-07T09:13:26.1063341Z running benchmark:  70% 21/30 [00:03<00:01,  5.82it/s]
2025-09-07T09:13:26.2725725Z running benchmark:  73% 22/30 [00:03<00:01,  5.71it/s]
2025-09-07T09:13:26.4447964Z running benchmark:  77% 23/30 [00:03<00:01,  5.80it/s]
2025-09-07T09:13:26.6133178Z running benchmark:  80% 24/30 [00:04<00:01,  5.80it/s]
2025-09-07T09:13:26.7829483Z running benchmark:  83% 25/30 [00:04<00:00,  5.84it/s]
2025-09-07T09:13:26.9542997Z running benchmark:  87% 26/30 [00:04<00:00,  5.86it/s]
2025-09-07T09:13:27.1233184Z running benchmark:  90% 27/30 [00:04<00:00,  5.85it/s]
2025-09-07T09:13:27.2935904Z running benchmark:  93% 28/30 [00:04<00:00,  5.87it/s]
2025-09-07T09:13:27.4633439Z running benchmark:  97% 29/30 [00:04<00:00,  5.87it/s]
2025-09-07T09:13:27.4634178Z running benchmark: 100% 30/30 [00:05<00:00,  5.88it/s]
2025-09-07T09:13:27.4634424Z running benchmark: 100% 30/30 [00:05<00:00,  5.87it/s]
2025-09-07T09:13:27.5963538Z 1.049x
2025-09-07T09:13:27.5967523Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:13:30.0970895Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:13:30.0971869Z   import pynvml  # type: ignore[import]
2025-09-07T09:13:32.1116838Z 
2025-09-07T09:13:34.5120343Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:13:34.5120627Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:13:34.5275984Z cpu  eval  hf_T5                              
2025-09-07T09:13:42.1168381Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1168722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1169123Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1169485Z     return mod(**inputs)
2025-09-07T09:13:42.1170171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1170582Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1170960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1171336Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1171696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1172061Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1172545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1172935Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1173308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1173712Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1174096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
2025-09-07T09:13:42.1174482Z     position_bias = position_bias + causal_mask
2025-09-07T09:13:42.1174630Z 
2025-09-07T09:13:42.1174738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1175170Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1175505Z     return mod(**inputs)
2025-09-07T09:13:42.1175871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1176258Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1176619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1176992Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1177351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1177776Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1178150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1178525Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1178891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1179292Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1179706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1180176Z     return self.weight * hidden_states
2025-09-07T09:13:42.1180320Z 
2025-09-07T09:13:42.1180427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1180862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1181192Z     return mod(**inputs)
2025-09-07T09:13:42.1181545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1181924Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1182288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1182659Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1183009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1183383Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1183753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1184128Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1184544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1184969Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1185339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1185725Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1185865Z 
2025-09-07T09:13:42.1185967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1186337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1186668Z     return mod(**inputs)
2025-09-07T09:13:42.1188373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1188774Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1189138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1189512Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1189866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1190220Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1190596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1190974Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1191345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1191715Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1192085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1192449Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1192581Z 
2025-09-07T09:13:42.1192690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1193096Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1193417Z     return mod(**inputs)
2025-09-07T09:13:42.1193761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1194129Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1194486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1194849Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1195194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1195603Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1195967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1196395Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1196754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1197128Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1197495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1197914Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1198107Z 
2025-09-07T09:13:42.1198213Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1198578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1198899Z     return mod(**inputs)
2025-09-07T09:13:42.1199237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1199618Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1200012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1200381Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1200716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1201085Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1201459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1201832Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1202244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1202626Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1203000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1203592Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1203806Z 
2025-09-07T09:13:42.1203909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1204270Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1204598Z     return mod(**inputs)
2025-09-07T09:13:42.1204949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1205329Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1205686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1206057Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1206405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1206779Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1207150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1207524Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1207890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1208295Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1208670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1209027Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1209303Z 
2025-09-07T09:13:42.1209412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1209766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1210150Z     return mod(**inputs)
2025-09-07T09:13:42.1210495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1210868Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1211227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1211598Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1211943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1212305Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1212674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1213073Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1213443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1213814Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1214233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1214638Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1214808Z 
2025-09-07T09:13:42.1214905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1215258Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1215580Z     return mod(**inputs)
2025-09-07T09:13:42.1215960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1216342Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1216700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1217065Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1217401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1217828Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1218199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1218573Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1218934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1219302Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1219670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1220076Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1220233Z 
2025-09-07T09:13:42.1220333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1220680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1220993Z     return mod(**inputs)
2025-09-07T09:13:42.1221335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1221707Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1222064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1222449Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1222783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1223219Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1223589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1224005Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1224366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1224740Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1225097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1225494Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1225645Z 
2025-09-07T09:13:42.1225747Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1226091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1226411Z     return mod(**inputs)
2025-09-07T09:13:42.1226755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1227143Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1227535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1227905Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1228247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1228601Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1228964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1229337Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1229730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1230111Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1230472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1230844Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1230970Z 
2025-09-07T09:13:42.1231068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1231414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1231737Z     return mod(**inputs)
2025-09-07T09:13:42.1232090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1232460Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1245087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1245563Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1245921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1246294Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1246685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1247094Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1247477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1247873Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1248242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1248601Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1248739Z 
2025-09-07T09:13:42.1248836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1249186Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1249561Z     return mod(**inputs)
2025-09-07T09:13:42.1249899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1250298Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1250658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1251046Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1251380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1251727Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1252079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1252451Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1252817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1253193Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1253551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1253960Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1254096Z 
2025-09-07T09:13:42.1254192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1254536Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1254853Z     return mod(**inputs)
2025-09-07T09:13:42.1255179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1255546Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1255935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1256318Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1256655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1257013Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1257375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1257792Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1258165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1258534Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1258907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1259278Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1259410Z 
2025-09-07T09:13:42.1259515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1259856Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1260170Z     return mod(**inputs)
2025-09-07T09:13:42.1260504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1260872Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1261225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1261582Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1261918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1262279Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1262647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1263068Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1263427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1263841Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1264213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1264634Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1264817Z 
2025-09-07T09:13:42.1264927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1265268Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1265585Z     return mod(**inputs)
2025-09-07T09:13:42.1265929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1266307Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1266659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1267023Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1267398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1267763Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1268131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1268535Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1268904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1269278Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1269676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1270125Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1270341Z 
2025-09-07T09:13:42.1270442Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1270801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1271125Z     return mod(**inputs)
2025-09-07T09:13:42.1271466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1271842Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1272207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1272577Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1272920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1273277Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1273647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1274021Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1274389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1274762Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1275121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1275558Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1275768Z 
2025-09-07T09:13:42.1275866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1276219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1276627Z     return mod(**inputs)
2025-09-07T09:13:42.1276967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1277336Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1277740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1278107Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1278436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1278796Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1279162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1279533Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1279896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1280264Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1280633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1281005Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1281134Z 
2025-09-07T09:13:42.1281283Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1281627Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1281945Z     return mod(**inputs)
2025-09-07T09:13:42.1282281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1282655Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1283021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1283437Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1283785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1284146Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1284520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1284887Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1285262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1285634Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1285999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1286400Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1286560Z 
2025-09-07T09:13:42.1286662Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1287022Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1287341Z     return mod(**inputs)
2025-09-07T09:13:42.1287687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1288059Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1288411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1288779Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1289117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1289471Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1289830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1290255Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1290620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1290989Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1291391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1291781Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1291942Z 
2025-09-07T09:13:42.1292038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1292386Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1292703Z     return mod(**inputs)
2025-09-07T09:13:42.1293041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1293401Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1293761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1294125Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1294457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1294845Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1295229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1295607Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1296001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1296407Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1296791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1297227Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1297384Z 
2025-09-07T09:13:42.1297531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1297876Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1298187Z     return mod(**inputs)
2025-09-07T09:13:42.1298519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1298875Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1299219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1299577Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1299906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1300305Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1300702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1301156Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1301562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1301935Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1302285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1302676Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1302804Z 
2025-09-07T09:13:42.1302878Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1303264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1303648Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1303955Z     return mod(**inputs)
2025-09-07T09:13:42.1304415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1304785Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1305134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1305593Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1305922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1306267Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1306623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1307032Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1307437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1307855Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1308229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1308621Z     return self.weight * hidden_states
2025-09-07T09:13:42.1308749Z 
2025-09-07T09:13:42.1308893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1309260Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1309574Z     return mod(**inputs)
2025-09-07T09:13:42.1309906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1310330Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1310711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1311076Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1311491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1311853Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1312251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1312630Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1313039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1313451Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1313853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1314218Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1314348Z 
2025-09-07T09:13:42.1314447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1314793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1315112Z     return mod(**inputs)
2025-09-07T09:13:42.1315491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1315915Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1316321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1316679Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1317056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1317411Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1317762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1318139Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1318592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1319014Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1319449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1319813Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1319985Z 
2025-09-07T09:13:42.1320109Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1320457Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1320766Z     return mod(**inputs)
2025-09-07T09:13:42.1321125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1321493Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1321850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1322245Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1322605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1322957Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1323351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1323757Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1324120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1324516Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1324939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1325336Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1325471Z 
2025-09-07T09:13:42.1325585Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1326299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1326674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1326990Z     return mod(**inputs)
2025-09-07T09:13:42.1327348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1327742Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1328090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1328446Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1328804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1329166Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1329550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1329916Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1330276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1330702Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1331081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1331481Z     return self.weight * hidden_states
2025-09-07T09:13:42.1331603Z 
2025-09-07T09:13:42.1331695Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1332067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1332376Z     return mod(**inputs)
2025-09-07T09:13:42.1332703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1333141Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1333510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1333909Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1334237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1334620Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1334970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1335384Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1335734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1336131Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1336493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1336891Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1337032Z 
2025-09-07T09:13:42.1337124Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1337536Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1337892Z     return mod(**inputs)
2025-09-07T09:13:42.1338224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1338612Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1338972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1339373Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1339735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1340119Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1340483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1340879Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1341242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1341609Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1341997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1342360Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1342525Z 
2025-09-07T09:13:42.1342619Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1342960Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1343300Z     return mod(**inputs)
2025-09-07T09:13:42.1343628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1344021Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1344371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1344731Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1345089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1345440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1345791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1346157Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1346552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1346993Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1347351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1347810Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1347988Z 
2025-09-07T09:13:42.1348086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1348419Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1348749Z     return mod(**inputs)
2025-09-07T09:13:42.1349116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1349509Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1349881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1350249Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1350574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1350958Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1351358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1351760Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1352126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1352536Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1352893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1353332Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1353604Z 
2025-09-07T09:13:42.1353710Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1354054Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1354407Z     return mod(**inputs)
2025-09-07T09:13:42.1354755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1355125Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1355474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1355880Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1356261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1356612Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1356971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1357377Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1357767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1358170Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1358558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1358989Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1359191Z 
2025-09-07T09:13:42.1359286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1359654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1359965Z     return mod(**inputs)
2025-09-07T09:13:42.1360330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1360735Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1361084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1361519Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1361856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1362215Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1362578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1362948Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1363335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1363709Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1364070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1364441Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1364584Z 
2025-09-07T09:13:42.1364679Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1365089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1365436Z     return mod(**inputs)
2025-09-07T09:13:42.1365783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1366152Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1366514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1366883Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1367213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1367610Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1367982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1368366Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1368738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1369105Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1369479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1369880Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1370040Z 
2025-09-07T09:13:42.1370147Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1370498Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1370819Z     return mod(**inputs)
2025-09-07T09:13:42.1371174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1371561Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1371926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1372300Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1372670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1373050Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1373416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1373794Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1374153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1374575Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1374958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1375405Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1375562Z 
2025-09-07T09:13:42.1375666Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1376033Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1376366Z     return mod(**inputs)
2025-09-07T09:13:42.1376721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1377098Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1377447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1377916Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1378258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1378630Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1379018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1380028Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1380423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1380822Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1381215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1381625Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1381829Z 
2025-09-07T09:13:42.1381948Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1382395Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1382719Z     return mod(**inputs)
2025-09-07T09:13:42.1383080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1383450Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1383810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1384225Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1384562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1384933Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1385345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1385772Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1386176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1386601Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1387006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1387446Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1387604Z 
2025-09-07T09:13:42.1387720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1388117Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1388466Z     return mod(**inputs)
2025-09-07T09:13:42.1388791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1389153Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1389502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1389915Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1390242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1390653Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1391017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1391400Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1391778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1392156Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1392536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1392912Z     return self.weight * hidden_states
2025-09-07T09:13:42.1393042Z 
2025-09-07T09:13:42.1393144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1393484Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1393801Z     return mod(**inputs)
2025-09-07T09:13:42.1394166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1394539Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1394894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1395251Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1395582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1395928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1396330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1396712Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1397188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1397610Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1398016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1398393Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1398520Z 
2025-09-07T09:13:42.1398622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1398959Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1399278Z     return mod(**inputs)
2025-09-07T09:13:42.1399619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1399985Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1400335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1400703Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1401040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1401393Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1401761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1402137Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1402510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1402907Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1403486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1403852Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1403988Z 
2025-09-07T09:13:42.1404139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1404484Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1404798Z     return mod(**inputs)
2025-09-07T09:13:42.1405130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1405492Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1405847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1406210Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1406545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1406901Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1407253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1407626Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1408052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1408462Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1408855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1409222Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1409359Z 
2025-09-07T09:13:42.1409435Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1409659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1410053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1410370Z     return mod(**inputs)
2025-09-07T09:13:42.1410705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1411070Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1411426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1411784Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1412115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1412468Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1412827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1413194Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1413556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1413953Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1414338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1414743Z     return self.weight * hidden_states
2025-09-07T09:13:42.1414869Z 
2025-09-07T09:13:42.1414969Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1415306Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1415623Z     return mod(**inputs)
2025-09-07T09:13:42.1415953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1416318Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1416666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1417091Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1417425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1417870Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1418241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1418606Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1418968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1419337Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1419702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1420062Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1420204Z 
2025-09-07T09:13:42.1420301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1420648Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1420966Z     return mod(**inputs)
2025-09-07T09:13:42.1421381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1421747Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1422099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1422463Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1422790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1423141Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1423527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1423927Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1424293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1424694Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1425065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1425445Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1425575Z 
2025-09-07T09:13:42.1425666Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1426004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1426314Z     return mod(**inputs)
2025-09-07T09:13:42.1426637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1427037Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1427429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1427825Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1428168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1428522Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1428884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1429259Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1429619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1429980Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1430344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1430807Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1430984Z 
2025-09-07T09:13:42.1431088Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1431465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1431776Z     return mod(**inputs)
2025-09-07T09:13:42.1432107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1432472Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1432825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1433182Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1433514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1433867Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1434229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1434598Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1434987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1435366Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1435728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1436171Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1436377Z 
2025-09-07T09:13:42.1436478Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1436814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1437216Z     return mod(**inputs)
2025-09-07T09:13:42.1437553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1437916Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1438264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1438627Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1438958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1439312Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1439673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1440040Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1440401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1440775Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1441138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1441574Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1441779Z 
2025-09-07T09:13:42.1441875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1442226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1442543Z     return mod(**inputs)
2025-09-07T09:13:42.1442881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1443239Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1443591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1444016Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1444356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1444707Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1445097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1445465Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1445825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1446200Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1446568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1446928Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1447065Z 
2025-09-07T09:13:42.1447162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1447514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1447846Z     return mod(**inputs)
2025-09-07T09:13:42.1448167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1448566Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1448924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1449294Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1449628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1449978Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1450345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1450753Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1451114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1451475Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1451839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1452239Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1452396Z 
2025-09-07T09:13:42.1452496Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1452842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1453148Z     return mod(**inputs)
2025-09-07T09:13:42.1453488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1453866Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1454221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1454583Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1454911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1455266Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1455627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1455993Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1456345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1456713Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1457068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1457460Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1457687Z 
2025-09-07T09:13:42.1457787Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1458123Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1458472Z     return mod(**inputs)
2025-09-07T09:13:42.1458808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1459172Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1459516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1459877Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1460211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1460562Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1460918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1461280Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1461641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1462060Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1462432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1462824Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1462978Z 
2025-09-07T09:13:42.1463071Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1463420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1463736Z     return mod(**inputs)
2025-09-07T09:13:42.1464102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1464469Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1464820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1465186Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1465521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1465877Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1466232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1466599Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1466958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1467324Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1467688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1468050Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1468181Z 
2025-09-07T09:13:42.1468256Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1468480Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1468822Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1469128Z     return mod(**inputs)
2025-09-07T09:13:42.1469462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1469826Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1470179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1470540Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1470864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1471265Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1471628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1472057Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1472430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1472817Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1473197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1473563Z     return self.weight * hidden_states
2025-09-07T09:13:42.1473687Z 
2025-09-07T09:13:42.1473787Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1474123Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1474444Z     return mod(**inputs)
2025-09-07T09:13:42.1474775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1475145Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1475533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1475912Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1476257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1476610Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1476970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1477369Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1477776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1478197Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1478603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1478978Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1479104Z 
2025-09-07T09:13:42.1479197Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1479536Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1479851Z     return mod(**inputs)
2025-09-07T09:13:42.1480184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1480544Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1480899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1481266Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1481604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1481960Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1482309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1482688Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1483057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1483459Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1483856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1484220Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1484398Z 
2025-09-07T09:13:42.1484493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1484837Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1485189Z     return mod(**inputs)
2025-09-07T09:13:42.1485515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1485881Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1486230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1486598Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1486929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1487275Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1487634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1488014Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1488384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1488815Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1489207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1489576Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1489711Z 
2025-09-07T09:13:42.1489785Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1490010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1490343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1490658Z     return mod(**inputs)
2025-09-07T09:13:42.1491033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1491408Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1491761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1492122Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1492454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1492808Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1493168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1493530Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1493899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1494330Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1494712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1495084Z     return self.weight * hidden_states
2025-09-07T09:13:42.1495210Z 
2025-09-07T09:13:42.1495305Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1495649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1495961Z     return mod(**inputs)
2025-09-07T09:13:42.1496290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1496653Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1497000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1497358Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1497764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1498207Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1498562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1498985Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1499352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1499723Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1500090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1500450Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1500587Z 
2025-09-07T09:13:42.1500681Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1501032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1501354Z     return mod(**inputs)
2025-09-07T09:13:42.1501677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1502046Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1502429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1502798Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1503266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1503664Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1504042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1504420Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1504838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1505223Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1505581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1505950Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1506088Z 
2025-09-07T09:13:42.1506190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1506540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1506851Z     return mod(**inputs)
2025-09-07T09:13:42.1507188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1521156Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1521653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1522071Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1522443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1522849Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1523273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1523658Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1524027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1524396Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1524753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1525163Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1525341Z 
2025-09-07T09:13:42.1525543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1525902Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1526221Z     return mod(**inputs)
2025-09-07T09:13:42.1526618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1526988Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1527341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1527701Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1528042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1528402Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1528757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1529128Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1529488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1529854Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1530262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1530703Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1530912Z 
2025-09-07T09:13:42.1531012Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1531360Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1531677Z     return mod(**inputs)
2025-09-07T09:13:42.1532012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1532412Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1532765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1533120Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1533459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1533806Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1534165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1534529Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1534881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1535244Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1535595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1536026Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1536227Z 
2025-09-07T09:13:42.1536322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1536667Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1536976Z     return mod(**inputs)
2025-09-07T09:13:42.1537298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1537738Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1538086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1538443Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1538764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1539181Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1539538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1539901Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1540303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1540667Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1541028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1541393Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1541523Z 
2025-09-07T09:13:42.1541625Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1541961Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1542275Z     return mod(**inputs)
2025-09-07T09:13:42.1542612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1542977Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1543330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1543722Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1544056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1544407Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1544770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1545135Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1545486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1545888Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1546252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1546653Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1546821Z 
2025-09-07T09:13:42.1546925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1547256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1547565Z     return mod(**inputs)
2025-09-07T09:13:42.1547894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1548253Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1548600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1548956Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1549285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1549635Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1549986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1550344Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1550694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1551055Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1551408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1551785Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1551936Z 
2025-09-07T09:13:42.1552027Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1552403Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1552708Z     return mod(**inputs)
2025-09-07T09:13:42.1553029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1553420Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1553767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1554118Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1554442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1554780Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1555124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1555484Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1555834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1556190Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1556543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1556957Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1557108Z 
2025-09-07T09:13:42.1557198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1557533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1557837Z     return mod(**inputs)
2025-09-07T09:13:42.1558157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1558512Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1558889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1559248Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1559570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1559906Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1560259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1560614Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1560965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1561319Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1561666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1562022Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1562145Z 
2025-09-07T09:13:42.1562236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1562567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1562866Z     return mod(**inputs)
2025-09-07T09:13:42.1563190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1563541Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1563884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1564234Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1564550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1564890Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1565244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1565647Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1566001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:13:42.1566658Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:13:42.1566844Z 
2025-09-07T09:13:42.1566921Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1567029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1567254Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1567311Z     return mod(**inputs)
2025-09-07T09:13:42.1567571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1567638Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1567882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1567948Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1568167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1568242Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1568506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1568594Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1568828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1568913Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1569142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1569242Z     return self.weight * hidden_states
2025-09-07T09:13:42.1569249Z 
2025-09-07T09:13:42.1569348Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1569546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1569611Z     return mod(**inputs)
2025-09-07T09:13:42.1569844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1569908Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1570137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1570198Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1570412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1570480Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1570711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1570792Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1571020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1571132Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1571358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1571432Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1571435Z 
2025-09-07T09:13:42.1571526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1571714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1571769Z     return mod(**inputs)
2025-09-07T09:13:42.1572001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1572100Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1572331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1572428Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1572643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1572711Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1572937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1573014Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1573242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1573341Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1573571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1573642Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1573645Z 
2025-09-07T09:13:42.1573733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1573964Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1574019Z     return mod(**inputs)
2025-09-07T09:13:42.1574251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1574312Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1574539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1574600Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1574856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1574926Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1575149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1575226Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1575452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1575550Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1575777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1575844Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1575847Z 
2025-09-07T09:13:42.1575921Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1576007Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1576191Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1576249Z     return mod(**inputs)
2025-09-07T09:13:42.1576475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1576539Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1576769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1576828Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1577041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1577105Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1577337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1577406Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1577728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1577829Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1578100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1578173Z     return self.weight * hidden_states
2025-09-07T09:13:42.1578177Z 
2025-09-07T09:13:42.1578272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1578464Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1578520Z     return mod(**inputs)
2025-09-07T09:13:42.1578750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1578816Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1579046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1579110Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1579318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1579386Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1579650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1579721Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1579951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1580023Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1580250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1580317Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1580355Z 
2025-09-07T09:13:42.1580448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1580634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1580693Z     return mod(**inputs)
2025-09-07T09:13:42.1580927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1580988Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1581217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1581278Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1581488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1581555Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1581781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1581851Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1582077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1582149Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1582378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1582444Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1582448Z 
2025-09-07T09:13:42.1582536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1582720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1582774Z     return mod(**inputs)
2025-09-07T09:13:42.1583003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1583100Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1583332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1583392Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1583640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1583708Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1583935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1584006Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1584230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1584301Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1584526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1584645Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1584649Z 
2025-09-07T09:13:42.1584739Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1584920Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1585009Z     return mod(**inputs)
2025-09-07T09:13:42.1585242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1585303Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1585531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1585590Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1585804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1585906Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1586136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1586205Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1586433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1586505Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1586729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1586877Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1586881Z 
2025-09-07T09:13:42.1586968Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1587150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1587208Z     return mod(**inputs)
2025-09-07T09:13:42.1587438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1587500Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1587730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1587789Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1588001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1588064Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1588291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1588358Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1588585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1588688Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1588912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1589083Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1589086Z 
2025-09-07T09:13:42.1589175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1589360Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1589414Z     return mod(**inputs)
2025-09-07T09:13:42.1589643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1589705Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1589931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1589995Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1590208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1590272Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1590535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1590603Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1590829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1590897Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1591123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1591189Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1591192Z 
2025-09-07T09:13:42.1591314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1591501Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1591555Z     return mod(**inputs)
2025-09-07T09:13:42.1591787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1591850Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1592076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1592138Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1592348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1592416Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1592642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1592709Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1592937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1593008Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1593238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1593336Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1593340Z 
2025-09-07T09:13:42.1593433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1593614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1593668Z     return mod(**inputs)
2025-09-07T09:13:42.1593902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1593968Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1594242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1594305Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1594514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1594620Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1594847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1594921Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1595145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1595220Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1595446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1595536Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1595542Z 
2025-09-07T09:13:42.1595635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1595818Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1595879Z     return mod(**inputs)
2025-09-07T09:13:42.1596162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1596228Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1596464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1596526Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1596743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1596808Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1597070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1597145Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1597371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1597448Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1597674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1597769Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1597772Z 
2025-09-07T09:13:42.1597861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1598041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1598103Z     return mod(**inputs)
2025-09-07T09:13:42.1598334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1598400Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1598630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1598692Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1598911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1598977Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1599205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1599274Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1599500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1599572Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1599905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1599989Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1599992Z 
2025-09-07T09:13:42.1600094Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1600190Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1600373Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1600430Z     return mod(**inputs)
2025-09-07T09:13:42.1600662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1600722Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1600958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1601018Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1601229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1601302Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1601527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1601650Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1601881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1601969Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1602196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1602269Z     return self.weight * hidden_states
2025-09-07T09:13:42.1602276Z 
2025-09-07T09:13:42.1602381Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1602607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1602673Z     return mod(**inputs)
2025-09-07T09:13:42.1602901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1602966Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1603398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1603460Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1603679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1603745Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1603971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1604051Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1604279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1604390Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1604616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1604691Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1604695Z 
2025-09-07T09:13:42.1604785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1604965Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1605028Z     return mod(**inputs)
2025-09-07T09:13:42.1605256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1605322Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1605553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1605671Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1605885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1606001Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1606235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1606311Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1606544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1606651Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1606874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1606949Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1606954Z 
2025-09-07T09:13:42.1607041Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1607229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1607289Z     return mod(**inputs)
2025-09-07T09:13:42.1607568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1607636Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1607867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1607934Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1608148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1608217Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1608499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1608581Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1608811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1608917Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1609149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1609218Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1609222Z 
2025-09-07T09:13:42.1609292Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1609387Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1609573Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1609633Z     return mod(**inputs)
2025-09-07T09:13:42.1609868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1609933Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1610170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1610234Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1610452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1610521Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1610750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1610825Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1611054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1611155Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1611416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1611490Z     return self.weight * hidden_states
2025-09-07T09:13:42.1611523Z 
2025-09-07T09:13:42.1611616Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1611799Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1611862Z     return mod(**inputs)
2025-09-07T09:13:42.1612089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1612156Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1612386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1612449Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1612666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1612734Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1612967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1613075Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1613305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1613382Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1613609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1613684Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1613688Z 
2025-09-07T09:13:42.1613778Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1613993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1614054Z     return mod(**inputs)
2025-09-07T09:13:42.1614290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1614361Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1614590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1614652Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1614865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1614931Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1615160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1615232Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1615463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1615535Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1615764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1615839Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1615844Z 
2025-09-07T09:13:42.1615932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1616120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1616178Z     return mod(**inputs)
2025-09-07T09:13:42.1616414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1616476Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1616709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1616808Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1617020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1617086Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1617362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1617431Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1617737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1617808Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1618071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1618194Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1618198Z 
2025-09-07T09:13:42.1618299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1618494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1618550Z     return mod(**inputs)
2025-09-07T09:13:42.1618835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1618900Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1619133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1619195Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1619408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1619477Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1619738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1619811Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1620039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1620113Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1620348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1620498Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1620502Z 
2025-09-07T09:13:42.1620593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1620778Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1620835Z     return mod(**inputs)
2025-09-07T09:13:42.1621069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1621133Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1621362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1621421Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1621639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1621704Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1621927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1621997Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1622221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1622293Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1622519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1622686Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1622695Z 
2025-09-07T09:13:42.1622789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1623010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1623069Z     return mod(**inputs)
2025-09-07T09:13:42.1623297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1623363Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1623597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1623659Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1623879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1623948Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1624180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1624245Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1624507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1624583Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1624808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1624879Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1624883Z 
2025-09-07T09:13:42.1624971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1625157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1625213Z     return mod(**inputs)
2025-09-07T09:13:42.1625476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1625547Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1625778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1625845Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1626058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1626122Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1626354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1626424Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1626656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1626729Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1626956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1627063Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1627068Z 
2025-09-07T09:13:42.1627160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1627347Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1627403Z     return mod(**inputs)
2025-09-07T09:13:42.1627638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1627700Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1627928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1627995Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1628245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1628319Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1628545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1628650Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1628879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1628949Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1629177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1629267Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1629271Z 
2025-09-07T09:13:42.1629362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1629550Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1629605Z     return mod(**inputs)
2025-09-07T09:13:42.1629837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1629901Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1630170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1630231Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1630443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1630512Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1630742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1630816Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1631077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1631151Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1631384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1631485Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1631488Z 
2025-09-07T09:13:42.1631583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1631764Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1631825Z     return mod(**inputs)
2025-09-07T09:13:42.1632056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1632115Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1632350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1632413Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1632627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1632695Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1632921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1632996Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1633223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1633298Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1633521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1633590Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1633642Z 
2025-09-07T09:13:42.1633737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1633921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1634024Z     return mod(**inputs)
2025-09-07T09:13:42.1634256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1634321Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1634554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1634613Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1634830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1634895Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1635125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1635194Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1635420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:13:42.1635584Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:13:42.1635588Z 
2025-09-07T09:13:42.1635659Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1635752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1635935Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1635993Z     return mod(**inputs)
2025-09-07T09:13:42.1636224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1636287Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1636554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1636616Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1636833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1636900Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1637129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1637214Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1637439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1637529Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1637754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1637822Z     return self.weight * hidden_states
2025-09-07T09:13:42.1637827Z 
2025-09-07T09:13:42.1637919Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1638104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1638167Z     return mod(**inputs)
2025-09-07T09:13:42.1638395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1638457Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1638690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1638750Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1638967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1639033Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1639267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1639488Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1639715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1639865Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1640092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1640162Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1640165Z 
2025-09-07T09:13:42.1640255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1640438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1640497Z     return mod(**inputs)
2025-09-07T09:13:42.1640727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1640797Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1641023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1641091Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1641337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1641412Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1641649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1641726Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1641960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1642059Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1642321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1642396Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1642399Z 
2025-09-07T09:13:42.1642494Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1642682Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1642737Z     return mod(**inputs)
2025-09-07T09:13:42.1642970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1643036Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1643267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1643333Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1643545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1643615Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1643843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1643918Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1644147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1644244Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1644475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1644542Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1644545Z 
2025-09-07T09:13:42.1644614Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1644708Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1644888Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1644986Z     return mod(**inputs)
2025-09-07T09:13:42.1645216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:13:42.1645315Z     encoder_outputs = self.encoder(
2025-09-07T09:13:42.1645551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward
2025-09-07T09:13:42.1645649Z     hidden_states = self.final_layer_norm(hidden_states)
2025-09-07T09:13:42.1645879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1645944Z     return self.weight * hidden_states
2025-09-07T09:13:42.1645947Z 
2025-09-07T09:13:42.1646038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1646222Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1646278Z     return mod(**inputs)
2025-09-07T09:13:42.1646511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1646572Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1646840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1646900Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1647112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1647183Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1647410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1647484Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1647744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1647822Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1648053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1648121Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1648126Z 
2025-09-07T09:13:42.1648219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1648402Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1648462Z     return mod(**inputs)
2025-09-07T09:13:42.1648690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1648753Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1648984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1649048Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1649268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1649332Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1649560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1649632Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1649859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1649933Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1650162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1650286Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1650290Z 
2025-09-07T09:13:42.1650415Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1650594Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1650655Z     return mod(**inputs)
2025-09-07T09:13:42.1650923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1650991Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1651224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1651286Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1651502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1651570Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1651800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1651873Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1652098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1652179Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1652441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1652589Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1652593Z 
2025-09-07T09:13:42.1652687Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1652882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1652937Z     return mod(**inputs)
2025-09-07T09:13:42.1653171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1653272Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1653511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1653576Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1653790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1653856Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1654084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1654152Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1654381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1654452Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1654685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1654755Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1654758Z 
2025-09-07T09:13:42.1654846Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1655033Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1655092Z     return mod(**inputs)
2025-09-07T09:13:42.1655336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1655396Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1655626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1655690Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1655899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1655972Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1656236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1656306Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1656576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1656646Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1656877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1656972Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1656975Z 
2025-09-07T09:13:42.1657071Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1657255Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1657310Z     return mod(**inputs)
2025-09-07T09:13:42.1657599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1657661Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1657893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1657992Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1658208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1658280Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1658507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1658578Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1658805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1658927Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1659160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1659254Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1659260Z 
2025-09-07T09:13:42.1659353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1659534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1659595Z     return mod(**inputs)
2025-09-07T09:13:42.1659832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1659893Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1660130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1660190Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1660411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1660475Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1660701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1660776Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1660999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1661073Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1661296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1661388Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1661394Z 
2025-09-07T09:13:42.1661481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1661663Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1661761Z     return mod(**inputs)
2025-09-07T09:13:42.1661992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1662095Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1662327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1662386Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1662603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1662668Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1662902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1662972Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1663202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1663283Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1663509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1663619Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1663622Z 
2025-09-07T09:13:42.1663695Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1663789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1663972Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1664028Z     return mod(**inputs)
2025-09-07T09:13:42.1664264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1664327Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1664598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1664660Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1664875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1664951Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1665180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1665263Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1665489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1665573Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1665803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1665874Z     return self.weight * hidden_states
2025-09-07T09:13:42.1665877Z 
2025-09-07T09:13:42.1665969Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1666149Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1666208Z     return mod(**inputs)
2025-09-07T09:13:42.1666439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1666501Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1666735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1666796Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1667011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1667075Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1667341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1667420Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1667646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1667790Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1668016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1668085Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1668094Z 
2025-09-07T09:13:42.1668182Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1668364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1668424Z     return mod(**inputs)
2025-09-07T09:13:42.1668656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1668721Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1668952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1669013Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1669263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1669331Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1669561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1669636Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1669860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1670000Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1670230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1670305Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1670310Z 
2025-09-07T09:13:42.1670398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1670585Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1670640Z     return mod(**inputs)
2025-09-07T09:13:42.1670868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1670934Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1671161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1671223Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1671439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1671515Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1671755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1671837Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1672067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1672164Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1672391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1672460Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1672463Z 
2025-09-07T09:13:42.1672549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1672738Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1672829Z     return mod(**inputs)
2025-09-07T09:13:42.1673066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1673162Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1673394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1673457Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1673667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1673735Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1673961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1674032Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1674263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1674361Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1674591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1674691Z     return self.weight * hidden_states
2025-09-07T09:13:42.1674695Z 
2025-09-07T09:13:42.1674788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1674967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1675024Z     return mod(**inputs)
2025-09-07T09:13:42.1675259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1675319Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1675585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1675654Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1675866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1675939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1676168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1676241Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1676467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1676538Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1676765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1676832Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1676836Z 
2025-09-07T09:13:42.1676930Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1677111Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1677169Z     return mod(**inputs)
2025-09-07T09:13:42.1677401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1677462Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1677696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1677757Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1677974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1678039Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1678265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1678384Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1678612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1678727Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1678958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1679025Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1679029Z 
2025-09-07T09:13:42.1679126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1679309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1679373Z     return mod(**inputs)
2025-09-07T09:13:42.1679604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1679672Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1679905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1679964Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1680185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1680304Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1680541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1680612Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1680838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1680911Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1681179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1681305Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1681309Z 
2025-09-07T09:13:42.1681396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1681582Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1681643Z     return mod(**inputs)
2025-09-07T09:13:42.1681875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1681938Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1682169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1682237Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1682448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1682514Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1682749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1682817Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1683052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1683120Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1683346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1683497Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1683501Z 
2025-09-07T09:13:42.1683585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1683771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1683826Z     return mod(**inputs)
2025-09-07T09:13:42.1684109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1684172Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1684401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1684498Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1684713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1684788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1685015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1685083Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1685314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1685388Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1685621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1685688Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1685693Z 
2025-09-07T09:13:42.1685817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1686007Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1686063Z     return mod(**inputs)
2025-09-07T09:13:42.1686301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1686364Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1686597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1686656Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1686968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1687039Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1687267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1687345Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1687572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1687642Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1687874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1687970Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1687973Z 
2025-09-07T09:13:42.1688062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1688245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1688304Z     return mod(**inputs)
2025-09-07T09:13:42.1688537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1688600Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1688837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1688896Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1689111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1689177Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1689419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1689497Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1689762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1689836Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1690063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1690195Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1690198Z 
2025-09-07T09:13:42.1690294Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1690476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1690542Z     return mod(**inputs)
2025-09-07T09:13:42.1690771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1690838Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1691067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1691130Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1691345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1691412Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1691683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1691754Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1691981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1692057Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1692283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1692412Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1692417Z 
2025-09-07T09:13:42.1692508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1692694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1692759Z     return mod(**inputs)
2025-09-07T09:13:42.1692991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1693056Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1693286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1693356Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1693568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1693632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1693870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1693944Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1694174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1694246Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1694473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1694546Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1694549Z 
2025-09-07T09:13:42.1694620Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1694718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1694898Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1694953Z     return mod(**inputs)
2025-09-07T09:13:42.1695189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1695286Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1695518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1695609Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1695828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1695896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1696120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1696195Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1696422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:13:42.1696524Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1696755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1696821Z     return self.weight * hidden_states
2025-09-07T09:13:42.1696824Z 
2025-09-07T09:13:42.1696920Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1697138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1697200Z     return mod(**inputs)
2025-09-07T09:13:42.1697430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1697552Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1697784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1697843Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1698098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1698166Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1698397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1698467Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1698697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1698776Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1699000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1699070Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1699073Z 
2025-09-07T09:13:42.1699163Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1699345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1699411Z     return mod(**inputs)
2025-09-07T09:13:42.1699637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1699701Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1699934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1699993Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1700204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1700270Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1700498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1700566Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1700795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1700903Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1701131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1701247Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1701250Z 
2025-09-07T09:13:42.1701340Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1701524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1701580Z     return mod(**inputs)
2025-09-07T09:13:42.1701806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1701873Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1702100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1702169Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1702381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1702446Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1702717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1702785Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1703016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1703236Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1703469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1703584Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1703587Z 
2025-09-07T09:13:42.1703743Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1703945Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1704003Z     return mod(**inputs)
2025-09-07T09:13:42.1704247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1704312Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1704544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1704615Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1704833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1704909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1705141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1705220Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1705450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1705522Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1705757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1705902Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1705906Z 
2025-09-07T09:13:42.1706009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1706199Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1706254Z     return mod(**inputs)
2025-09-07T09:13:42.1706490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1706555Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1706860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1706926Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1707196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1707268Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1707502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1707580Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1707809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1707887Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1708117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1708190Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1708193Z 
2025-09-07T09:13:42.1708294Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1708485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1708607Z     return mod(**inputs)
2025-09-07T09:13:42.1708848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1708914Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1709155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1709217Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1709440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1709664Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1709900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1709982Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1710219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1710298Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1710525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1710634Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1710637Z 
2025-09-07T09:13:42.1710728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1710914Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1710980Z     return mod(**inputs)
2025-09-07T09:13:42.1711215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1711283Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1711510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1711574Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1711792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1711859Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1712087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1712155Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1712386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1712456Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1712717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1712813Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1712847Z 
2025-09-07T09:13:42.1712935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1713121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1713175Z     return mod(**inputs)
2025-09-07T09:13:42.1713406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1713472Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1713703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1713765Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1713976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1714041Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1714272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1714377Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1714608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1714679Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1714907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1714997Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1715000Z 
2025-09-07T09:13:42.1715088Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1715307Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1715365Z     return mod(**inputs)
2025-09-07T09:13:42.1715604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1715667Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1715900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1715966Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1716181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1716253Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1716482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1716553Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1716791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1716863Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1717098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1717173Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1717176Z 
2025-09-07T09:13:42.1717254Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1717344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1717532Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1717596Z     return mod(**inputs)
2025-09-07T09:13:42.1717827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1717896Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1718127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1718227Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1718451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1718554Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1718793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1718876Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1719105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1719200Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1719428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1719504Z     return self.weight * hidden_states
2025-09-07T09:13:42.1719507Z 
2025-09-07T09:13:42.1719596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1719789Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1719849Z     return mod(**inputs)
2025-09-07T09:13:42.1720117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1720191Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1720423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1720490Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1720705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1720774Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1721042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1721126Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1721362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1721473Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1721708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1721779Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1721783Z 
2025-09-07T09:13:42.1721869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1722059Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1722116Z     return mod(**inputs)
2025-09-07T09:13:42.1722353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1722415Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1722646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1722714Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1722928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1723001Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1723230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1723308Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1723541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1723638Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1723934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1724008Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1724045Z 
2025-09-07T09:13:42.1724137Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1724323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1724379Z     return mod(**inputs)
2025-09-07T09:13:42.1724611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1724671Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1724899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1724958Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1725169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1725240Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1725468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1725549Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1725811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1725914Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1726143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1726210Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1726214Z 
2025-09-07T09:13:42.1726285Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1726371Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1726588Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1726643Z     return mod(**inputs)
2025-09-07T09:13:42.1726875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1726947Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1727178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1727243Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1727455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1727522Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1727748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1727822Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1728053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1728149Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1728383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1728451Z     return self.weight * hidden_states
2025-09-07T09:13:42.1728454Z 
2025-09-07T09:13:42.1728540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1728724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1728778Z     return mod(**inputs)
2025-09-07T09:13:42.1729007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1729081Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1729315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1729418Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1729632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1729734Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1729965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1730038Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1730269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1730341Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1730571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1730638Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1730643Z 
2025-09-07T09:13:42.1730736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1730918Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1730975Z     return mod(**inputs)
2025-09-07T09:13:42.1731244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1731308Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1731541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1731600Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1731813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1731884Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1732140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1732216Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1732444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1732517Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1732751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1732817Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1732821Z 
2025-09-07T09:13:42.1732912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1733098Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1733162Z     return mod(**inputs)
2025-09-07T09:13:42.1733389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1733455Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1733685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1733745Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1733968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1734031Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1734255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1734323Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1734548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1734617Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1734846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1734994Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1735000Z 
2025-09-07T09:13:42.1735087Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1735304Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1735361Z     return mod(**inputs)
2025-09-07T09:13:42.1735589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1735651Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1735878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1735937Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1736148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1736214Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1736443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1736510Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1736774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1736845Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1737070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1737216Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1737219Z 
2025-09-07T09:13:42.1737305Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1737556Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1737644Z     return mod(**inputs)
2025-09-07T09:13:42.1737879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1737945Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1738178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1738242Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1738451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1738518Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1738752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1738820Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1739053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1739124Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1739348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1739418Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1739422Z 
2025-09-07T09:13:42.1739511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1739697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1739752Z     return mod(**inputs)
2025-09-07T09:13:42.1739984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1740047Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1740277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1740347Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1740601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1740680Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1740950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1741024Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1741260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1741335Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1741572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1741672Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1741675Z 
2025-09-07T09:13:42.1741780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1741975Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1742035Z     return mod(**inputs)
2025-09-07T09:13:42.1749311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1749476Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1749755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1749825Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1750080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1750156Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1750416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1750544Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1750800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1750880Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1751116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1751222Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1751227Z 
2025-09-07T09:13:42.1751330Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1751533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1751593Z     return mod(**inputs)
2025-09-07T09:13:42.1751835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1751908Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1752143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1752208Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1752430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1752506Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1752740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1752817Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1753046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1753120Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1753346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1753485Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1753490Z 
2025-09-07T09:13:42.1753583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1753779Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1753869Z     return mod(**inputs)
2025-09-07T09:13:42.1754108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1754172Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1754402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1754465Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1754677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1754746Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1754974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1755043Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1755270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1755377Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1755608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1755677Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1755680Z 
2025-09-07T09:13:42.1755771Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1755957Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1756018Z     return mod(**inputs)
2025-09-07T09:13:42.1756294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1756357Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1756589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1756651Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1756863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1756930Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1757155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1757223Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1757448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:13:42.1757570Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:13:42.1757579Z 
2025-09-07T09:13:42.1757650Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1757737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1757921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1757978Z     return mod(**inputs)
2025-09-07T09:13:42.1758211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1758271Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1758512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1758573Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1758791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1758860Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1759123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1759192Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1759420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:13:42.1759549Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1759778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1759844Z     return self.weight * hidden_states
2025-09-07T09:13:42.1759847Z 
2025-09-07T09:13:42.1759941Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1760124Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1760178Z     return mod(**inputs)
2025-09-07T09:13:42.1760412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1760475Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1760703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1760764Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1761012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1761081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1761308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1761379Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1761604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1761676Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1761941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1762010Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1762013Z 
2025-09-07T09:13:42.1762104Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1762288Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1762348Z     return mod(**inputs)
2025-09-07T09:13:42.1762578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1762643Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1762883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1762945Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1763163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1763229Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1763455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1763531Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1763756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1763829Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1764056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1764125Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1764128Z 
2025-09-07T09:13:42.1764220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1764404Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1764499Z     return mod(**inputs)
2025-09-07T09:13:42.1764732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1764798Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1765065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1765127Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1765344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1765410Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1765638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1765708Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1765937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1766014Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1766238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1766364Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1766409Z 
2025-09-07T09:13:42.1766497Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1766679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1766738Z     return mod(**inputs)
2025-09-07T09:13:42.1766970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1767040Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1767270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1767370Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1767586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1767653Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1767890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1767959Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1768192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1768264Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1768489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1768650Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1768655Z 
2025-09-07T09:13:42.1768750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1768942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1768999Z     return mod(**inputs)
2025-09-07T09:13:42.1769239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1769303Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1769529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1769599Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1769812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1769882Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1770114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1770219Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1770454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1770560Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1770796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1770869Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1770872Z 
2025-09-07T09:13:42.1770964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1771157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1771214Z     return mod(**inputs)
2025-09-07T09:13:42.1771451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1771515Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1771752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1771813Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1772024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1772153Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1772386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1772460Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1772685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1772758Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1773021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1773125Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1773129Z 
2025-09-07T09:13:42.1773222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1773405Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1773463Z     return mod(**inputs)
2025-09-07T09:13:42.1773701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1773761Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1773997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1774059Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1774276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1774343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1774575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1774651Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1774880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1774957Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1775186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1775280Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1775283Z 
2025-09-07T09:13:42.1775377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1775559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1775622Z     return mod(**inputs)
2025-09-07T09:13:42.1775894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1775959Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1776195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1776295Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1776516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1776583Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1776819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1776890Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1777116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1777197Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1777423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1777581Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1777586Z 
2025-09-07T09:13:42.1777709Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1777896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1777957Z     return mod(**inputs)
2025-09-07T09:13:42.1778188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1778254Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1778480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1778544Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1778787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1778855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1779088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1779161Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1779388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1779459Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1779704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1779775Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1779779Z 
2025-09-07T09:13:42.1779854Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1779964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1780160Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1780216Z     return mod(**inputs)
2025-09-07T09:13:42.1780465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1780539Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1780777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1780840Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1781058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1781138Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1781369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1781453Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1781723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1781814Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1782078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1782147Z     return self.weight * hidden_states
2025-09-07T09:13:42.1782150Z 
2025-09-07T09:13:42.1782245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1782433Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1782492Z     return mod(**inputs)
2025-09-07T09:13:42.1782725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1782787Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1783021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1783085Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1783297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1783474Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1783705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1783789Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1784015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1784123Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1784354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1784461Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1784464Z 
2025-09-07T09:13:42.1784557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1784743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1784805Z     return mod(**inputs)
2025-09-07T09:13:42.1785036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1785099Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1785331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1785392Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1785610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1785676Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1785904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1785982Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1786210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1786320Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1786545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1786614Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1786621Z 
2025-09-07T09:13:42.1786710Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1786892Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1786950Z     return mod(**inputs)
2025-09-07T09:13:42.1787179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1787278Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1787508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1787602Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1787822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1787887Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1788117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1788190Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1788415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1788519Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1788744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1788815Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1788820Z 
2025-09-07T09:13:42.1788889Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1789168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1789355Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1789412Z     return mod(**inputs)
2025-09-07T09:13:42.1789644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1789705Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1789939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1789997Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1790244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1790315Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1790542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1790618Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1790858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1790953Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1791184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1791249Z     return self.weight * hidden_states
2025-09-07T09:13:42.1791252Z 
2025-09-07T09:13:42.1791341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1791524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1791583Z     return mod(**inputs)
2025-09-07T09:13:42.1791811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1791877Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1792110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1792173Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1792386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1792451Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1792678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1792750Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1793015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1793093Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1793373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1793443Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1793448Z 
2025-09-07T09:13:42.1793540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1793724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1793786Z     return mod(**inputs)
2025-09-07T09:13:42.1794014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1794077Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1794311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1794377Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1794593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1794659Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1794924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1794996Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1795223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1795295Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1795522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1795591Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1795631Z 
2025-09-07T09:13:42.1795720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1795907Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1795964Z     return mod(**inputs)
2025-09-07T09:13:42.1796197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1796259Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1796486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1796549Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1796759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1796823Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1797052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1797122Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1797352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1797420Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1797645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1797768Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1797771Z 
2025-09-07T09:13:42.1797856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1798043Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1798096Z     return mod(**inputs)
2025-09-07T09:13:42.1798329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1798431Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1798661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1798755Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1798968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1799037Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1799262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1799331Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1799556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1799624Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1799854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1800003Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1800006Z 
2025-09-07T09:13:42.1800099Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1800311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1800368Z     return mod(**inputs)
2025-09-07T09:13:42.1800603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1800662Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1800892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1800953Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1801209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1801284Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1801509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1801583Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1801807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1801877Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1802103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1802167Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1802171Z 
2025-09-07T09:13:42.1802264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1802443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1802513Z     return mod(**inputs)
2025-09-07T09:13:42.1802740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1802806Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1803211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1803275Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1803485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1803551Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1803780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1803847Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1804075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1804224Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1804448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1804597Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1804600Z 
2025-09-07T09:13:42.1804690Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1804874Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1804929Z     return mod(**inputs)
2025-09-07T09:13:42.1805159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1805224Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1805450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1805514Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1805725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1805790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1806071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1806141Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1806370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1806438Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1806663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1806755Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1806758Z 
2025-09-07T09:13:42.1806892Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1807083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1807138Z     return mod(**inputs)
2025-09-07T09:13:42.1807372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1807438Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1807669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1807731Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1807946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1808017Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1808244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1808316Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1808546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1808615Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1808847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1808940Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1808943Z 
2025-09-07T09:13:42.1809032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1809212Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1809266Z     return mod(**inputs)
2025-09-07T09:13:42.1809496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1809555Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1809833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1809892Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1810105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1810203Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1810431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1810503Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1810733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1810808Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1811032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1811102Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1811106Z 
2025-09-07T09:13:42.1811178Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1811266Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1811455Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1811541Z     return mod(**inputs)
2025-09-07T09:13:42.1811776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1811838Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1812067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1812130Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1812340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1812443Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1812675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1812745Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1812976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:13:42.1813070Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1813299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1813367Z     return self.weight * hidden_states
2025-09-07T09:13:42.1813370Z 
2025-09-07T09:13:42.1813458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1813647Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1813701Z     return mod(**inputs)
2025-09-07T09:13:42.1813935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1813998Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1814225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1814292Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1814501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1814573Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1814799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1814871Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1815095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1815169Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1815452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1815519Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1815553Z 
2025-09-07T09:13:42.1815649Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1815839Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1815895Z     return mod(**inputs)
2025-09-07T09:13:42.1816132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1816195Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1816426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1816486Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1816700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1816776Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1817002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1817106Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1817334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1817410Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1817689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1817757Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1817760Z 
2025-09-07T09:13:42.1817853Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1818082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1818145Z     return mod(**inputs)
2025-09-07T09:13:42.1818377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1818439Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1818670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1818730Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1818943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1819007Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1819233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1819304Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1819531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1819604Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1819828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1819950Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1819953Z 
2025-09-07T09:13:42.1820040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1820220Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1820279Z     return mod(**inputs)
2025-09-07T09:13:42.1820507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1820569Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1820801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1820920Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1821135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1821241Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1821472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1821540Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1821765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1821834Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1822057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1822204Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1822209Z 
2025-09-07T09:13:42.1822296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1822485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1822541Z     return mod(**inputs)
2025-09-07T09:13:42.1822804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1822871Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1823100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1823163Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1823374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1823438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1823700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1823770Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1823999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1824070Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1824302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1824370Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1824373Z 
2025-09-07T09:13:42.1824459Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1824644Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1824699Z     return mod(**inputs)
2025-09-07T09:13:42.1824930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1824995Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1825223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1825290Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1825503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1825573Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1825801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1825870Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1826102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1826172Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1826402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1826534Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1826538Z 
2025-09-07T09:13:42.1826631Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1826848Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1826905Z     return mod(**inputs)
2025-09-07T09:13:42.1827140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1827201Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1827436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1827496Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1827710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1827783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1828008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1828086Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1828344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1828423Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1828651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1828742Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1828745Z 
2025-09-07T09:13:42.1828836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1829019Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1829114Z     return mod(**inputs)
2025-09-07T09:13:42.1829350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1829413Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1829652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1829711Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1829931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1829996Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1830223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1830303Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1830530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1830609Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1830836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1830938Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1830941Z 
2025-09-07T09:13:42.1831033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1831217Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1831280Z     return mod(**inputs)
2025-09-07T09:13:42.1831509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1831572Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1831799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1831860Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1832112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1832177Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1832449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1832517Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1832744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1832816Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1833041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1833109Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1833113Z 
2025-09-07T09:13:42.1833199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1833385Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1833440Z     return mod(**inputs)
2025-09-07T09:13:42.1833669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1833766Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1834002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1834065Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1834281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1834349Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1834576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1834694Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1834930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:13:42.1835054Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:13:42.1835059Z 
2025-09-07T09:13:42.1835131Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1835224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1835412Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1835470Z     return mod(**inputs)
2025-09-07T09:13:42.1835702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1835765Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1835995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1836056Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1836273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1836337Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1836570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1836651Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1836877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1836963Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1837188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1837255Z     return self.weight * hidden_states
2025-09-07T09:13:42.1837259Z 
2025-09-07T09:13:42.1837349Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1837575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1837630Z     return mod(**inputs)
2025-09-07T09:13:42.1837861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1837958Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1838188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1838253Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1838465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1838531Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1838765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1838842Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1839074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1839182Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1839441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1839517Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1839520Z 
2025-09-07T09:13:42.1839605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1839790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1839845Z     return mod(**inputs)
2025-09-07T09:13:42.1840079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1840139Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1840403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1840468Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1840680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1840753Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1840982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1841057Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1841289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1841388Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1841619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1841692Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1841696Z 
2025-09-07T09:13:42.1841789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1841970Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1842027Z     return mod(**inputs)
2025-09-07T09:13:42.1842262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1842324Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1842555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1842614Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1842826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1842898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1843165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1843243Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1843469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1843603Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1843834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1843901Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1843903Z 
2025-09-07T09:13:42.1843974Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1844059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1844252Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1844309Z     return mod(**inputs)
2025-09-07T09:13:42.1844546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1844610Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1844842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1844941Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1845156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1845225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1845457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1845526Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1845754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1845885Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1846113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1846182Z     return self.weight * hidden_states
2025-09-07T09:13:42.1846185Z 
2025-09-07T09:13:42.1846275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1846461Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1846516Z     return mod(**inputs)
2025-09-07T09:13:42.1846748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1846808Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1847037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1847102Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1847315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1847383Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1847614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1847686Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1847917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1847988Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1848217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1848283Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1848286Z 
2025-09-07T09:13:42.1848375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1848594Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1848648Z     return mod(**inputs)
2025-09-07T09:13:42.1848879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1848970Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1849202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1849262Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1849471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1849538Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1849765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1849835Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1850061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1850130Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1850362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1850464Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1850467Z 
2025-09-07T09:13:42.1850559Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1850741Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1850799Z     return mod(**inputs)
2025-09-07T09:13:42.1851029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1851089Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1851357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1851419Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1851635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1851702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1851928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1851998Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1852222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1852294Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1852520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1852641Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1852649Z 
2025-09-07T09:13:42.1852737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1852916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1852974Z     return mod(**inputs)
2025-09-07T09:13:42.1853201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1853264Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1853493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1853550Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1853760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1853825Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1854056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1854159Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1854384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1854491Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1854721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1854870Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1854874Z 
2025-09-07T09:13:42.1854962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1855149Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1855209Z     return mod(**inputs)
2025-09-07T09:13:42.1855443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1855508Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1855735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1855803Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1856061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1856126Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1856355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1856421Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1856647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1856713Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1856970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1857040Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1857044Z 
2025-09-07T09:13:42.1857129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1857315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1857369Z     return mod(**inputs)
2025-09-07T09:13:42.1857633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1857694Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1857924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1857987Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1858198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1858265Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1858489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1858556Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1858784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1858851Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1859077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1859172Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1859175Z 
2025-09-07T09:13:42.1859264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1859450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1859544Z     return mod(**inputs)
2025-09-07T09:13:42.1859783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1859850Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1860127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1860187Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1860396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1860468Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1860694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1860763Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1860993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1861066Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1861292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1861382Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1861385Z 
2025-09-07T09:13:42.1861511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1861694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1861749Z     return mod(**inputs)
2025-09-07T09:13:42.1861980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1862039Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1862268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1862369Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1862586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1862651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1862877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1862947Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1863171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1863240Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1863463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1863556Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1863559Z 
2025-09-07T09:13:42.1863649Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1863834Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1863893Z     return mod(**inputs)
2025-09-07T09:13:42.1864123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1864189Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1864419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1864477Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1864691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1864754Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1864979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1865048Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1865310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1865382Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1865645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1865713Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1865716Z 
2025-09-07T09:13:42.1865783Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1865871Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1866055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1866113Z     return mod(**inputs)
2025-09-07T09:13:42.1866346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1866410Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1866639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1866702Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1866946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1867016Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1867243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1867318Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1867542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:13:42.1867634Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1867895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1867961Z     return self.weight * hidden_states
2025-09-07T09:13:42.1867964Z 
2025-09-07T09:13:42.1868052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1868234Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1868289Z     return mod(**inputs)
2025-09-07T09:13:42.1868525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1868589Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1868820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1868879Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1869092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1869159Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1869386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1869461Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1869690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1869763Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1869989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1870054Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1870058Z 
2025-09-07T09:13:42.1870146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1870326Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1870383Z     return mod(**inputs)
2025-09-07T09:13:42.1870612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1870713Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1870943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1871098Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1871309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1871375Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1871602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1871670Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1871895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1871967Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1872192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1872258Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1872263Z 
2025-09-07T09:13:42.1872351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1872562Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1872620Z     return mod(**inputs)
2025-09-07T09:13:42.1872848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1872910Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1873138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1873197Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1873446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1873513Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1873742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1873814Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1874041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1874110Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1874334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1874450Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1874454Z 
2025-09-07T09:13:42.1874539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1874724Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1874781Z     return mod(**inputs)
2025-09-07T09:13:42.1875010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1875075Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1875303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1875365Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1875575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1875639Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1875865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1875934Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1876163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1876265Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1876490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1876679Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1876683Z 
2025-09-07T09:13:42.1876769Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1876954Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1877007Z     return mod(**inputs)
2025-09-07T09:13:42.1877240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1877298Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1877529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1877592Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1877801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1877869Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1878127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1878197Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1878424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1878493Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1878719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1878784Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1878820Z 
2025-09-07T09:13:42.1878909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1879089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1879145Z     return mod(**inputs)
2025-09-07T09:13:42.1879378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1879438Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1879668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1879725Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1879935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1880000Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1880223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1880294Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1880517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1880591Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1880815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1880910Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1880912Z 
2025-09-07T09:13:42.1881001Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1881182Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1881238Z     return mod(**inputs)
2025-09-07T09:13:42.1881468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1881561Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1881793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1881885Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1882099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1882163Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1882392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1882460Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1882684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1882757Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1882984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1883079Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1883082Z 
2025-09-07T09:13:42.1883171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1883384Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1883441Z     return mod(**inputs)
2025-09-07T09:13:42.1883673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1883734Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1883962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1884020Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1884232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1884334Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1884564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1884635Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1884862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1884933Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1885156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1885251Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1885254Z 
2025-09-07T09:13:42.1885341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1885524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1885581Z     return mod(**inputs)
2025-09-07T09:13:42.1885808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1885869Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1886101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1886161Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1886372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1886436Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1886663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1886730Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1886956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1887061Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1887289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1887387Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1887391Z 
2025-09-07T09:13:42.1887460Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1887549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1887730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1887788Z     return mod(**inputs)
2025-09-07T09:13:42.1888014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1888074Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1888303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1888363Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1888577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1888644Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1888901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1888984Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1889209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1889296Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1889520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1889588Z     return self.weight * hidden_states
2025-09-07T09:13:42.1889591Z 
2025-09-07T09:13:42.1889711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1889894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1889953Z     return mod(**inputs)
2025-09-07T09:13:42.1890190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1890255Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1890483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1890542Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1890755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1890819Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1891046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1891123Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1891347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1891456Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1891680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1891751Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1891754Z 
2025-09-07T09:13:42.1891841Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1892023Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1892078Z     return mod(**inputs)
2025-09-07T09:13:42.1892305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1892368Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1892635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1892698Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1892947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1893012Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1893239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1893318Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1893549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1893650Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1893883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1893956Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1893960Z 
2025-09-07T09:13:42.1894047Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1894237Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1894330Z     return mod(**inputs)
2025-09-07T09:13:42.1894565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1894625Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1894854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1894918Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1895127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1895233Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1895464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1895539Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1895771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1895871Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1896099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1896167Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1896170Z 
2025-09-07T09:13:42.1896265Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1896447Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1896502Z     return mod(**inputs)
2025-09-07T09:13:42.1896739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1896802Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1897036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1897097Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1897306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1897378Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1897663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1897738Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1897964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:13:42.1898143Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:13:42.1898146Z 
2025-09-07T09:13:42.1898214Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1898299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1898519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1898574Z     return mod(**inputs)
2025-09-07T09:13:42.1898807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1898867Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1899095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1899157Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1899365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1899434Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1899658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1899728Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1899991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:13:42.1900087Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1900316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1900380Z     return self.weight * hidden_states
2025-09-07T09:13:42.1900383Z 
2025-09-07T09:13:42.1900470Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1900650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1900735Z     return mod(**inputs)
2025-09-07T09:13:42.1900968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1901027Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1901259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1901318Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1901528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1901593Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1901818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1901887Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1902111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1902183Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1902410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1902478Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1902482Z 
2025-09-07T09:13:42.1902571Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1902751Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1902808Z     return mod(**inputs)
2025-09-07T09:13:42.1903036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1903244Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1903476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1903534Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1903842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1903910Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1904142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1904282Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1904507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1904580Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1904804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1904870Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1904875Z 
2025-09-07T09:13:42.1904969Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1905159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1905220Z     return mod(**inputs)
2025-09-07T09:13:42.1905453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1905519Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1905812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1905876Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1906093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1906162Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1906392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1906461Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1906745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1906821Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1907047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1907173Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1907177Z 
2025-09-07T09:13:42.1907267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1907451Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1907515Z     return mod(**inputs)
2025-09-07T09:13:42.1907745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1907813Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1908043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1908111Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1908322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1908391Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1908624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1908694Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1908926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1908996Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1909220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1909372Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1909446Z 
2025-09-07T09:13:42.1909536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1909728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1909824Z     return mod(**inputs)
2025-09-07T09:13:42.1910063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1910129Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1910360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1910425Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1910636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1910706Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1910935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1911004Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1911235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1911343Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1911578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1911645Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1911648Z 
2025-09-07T09:13:42.1911742Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1911924Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1911978Z     return mod(**inputs)
2025-09-07T09:13:42.1912250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1912316Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1912551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1912612Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1912827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1912899Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1913125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1913197Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1913424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1913493Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1913726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1913830Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1913833Z 
2025-09-07T09:13:42.1913928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1914112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1914172Z     return mod(**inputs)
2025-09-07T09:13:42.1914397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1914460Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1914693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1914754Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1914970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1915071Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1915299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1915406Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1915633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1915708Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1915934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1916025Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1916034Z 
2025-09-07T09:13:42.1916122Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1916305Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1916367Z     return mod(**inputs)
2025-09-07T09:13:42.1916596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1916658Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1916930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1916991Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1917210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1917274Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1917509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1917576Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1917858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1917935Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1918162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1918264Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1918267Z 
2025-09-07T09:13:42.1918357Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1918541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1918599Z     return mod(**inputs)
2025-09-07T09:13:42.1918827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1918892Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1919122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1919190Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1919400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1919467Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1919703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:13:42.1919771Z     self_attention_outputs = self.layer[0](
2025-09-07T09:13:42.1920001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:13:42.1920068Z     attention_output = self.SelfAttention(
2025-09-07T09:13:42.1920291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1920361Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1920364Z 
2025-09-07T09:13:42.1920434Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1920573Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1920757Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1920812Z     return mod(**inputs)
2025-09-07T09:13:42.1921087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1921151Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1921386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1921447Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1921662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1921729Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1921959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1922037Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1922262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:13:42.1922366Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1922637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1922707Z     return self.weight * hidden_states
2025-09-07T09:13:42.1922710Z 
2025-09-07T09:13:42.1922803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1922987Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1923050Z     return mod(**inputs)
2025-09-07T09:13:42.1923280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1923387Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1923622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1923685Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1923904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1923970Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1924201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1924272Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1924500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1924576Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1924802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:13:42.1924878Z     query_states = self.q(hidden_states)
2025-09-07T09:13:42.1924881Z 
2025-09-07T09:13:42.1924971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1925155Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1925218Z     return mod(**inputs)
2025-09-07T09:13:42.1925450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1925516Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1925745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1925810Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1926022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1926087Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1926365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1926440Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1926706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1926779Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1927004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:13:42.1927076Z     key_states = self.k(current_states)
2025-09-07T09:13:42.1927079Z 
2025-09-07T09:13:42.1927168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1927354Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1927409Z     return mod(**inputs)
2025-09-07T09:13:42.1927640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1927708Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1927938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1928005Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1928248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1928319Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1928544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1928612Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1928843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1928950Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1929188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:13:42.1929308Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:13:42.1929312Z 
2025-09-07T09:13:42.1929401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1929587Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1929642Z     return mod(**inputs)
2025-09-07T09:13:42.1929874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1929937Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1930165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1930231Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1930446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1930518Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1930742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1930819Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1931044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1931114Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1931344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:13:42.1931490Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:13:42.1931494Z 
2025-09-07T09:13:42.1931589Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1931776Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1932207Z     return mod(**inputs)
2025-09-07T09:13:42.1932451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1932554Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1932794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1932858Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1933076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1933146Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1933376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1933448Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1933676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1933751Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1933978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:13:42.1934081Z     value_states = self.v(current_states)
2025-09-07T09:13:42.1934085Z 
2025-09-07T09:13:42.1934178Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1934361Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1934420Z     return mod(**inputs)
2025-09-07T09:13:42.1934649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1934714Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1934982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1935045Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1935262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1935330Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1935560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1935629Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1935855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1935927Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1936151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1936252Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1936256Z 
2025-09-07T09:13:42.1936344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1936526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1936585Z     return mod(**inputs)
2025-09-07T09:13:42.1936814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1936878Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1937106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1937166Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1937380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1937445Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1937711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1937821Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1938048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1938152Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1938380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:13:42.1938476Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:13:42.1938480Z 
2025-09-07T09:13:42.1938567Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1938750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1938805Z     return mod(**inputs)
2025-09-07T09:13:42.1939034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1939099Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1939327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1939391Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1939653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1939721Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1939949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1940017Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1940243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1940315Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1940577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:13:42.1940670Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:13:42.1940673Z 
2025-09-07T09:13:42.1940761Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1940947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1941002Z     return mod(**inputs)
2025-09-07T09:13:42.1941235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1941296Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1941525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1941587Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1941799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1941867Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1942092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:13:42.1942164Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:13:42.1942388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:13:42.1942459Z     attention_output = self.EncDecAttention(
2025-09-07T09:13:42.1942686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:13:42.1942751Z     attn_output = self.o(attn_output)
2025-09-07T09:13:42.1942754Z 
2025-09-07T09:13:42.1942828Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1942914Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1943094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1943186Z     return mod(**inputs)
2025-09-07T09:13:42.1943417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1943516Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1943748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1943809Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1944022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1944087Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1944315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1944395Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1944624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:13:42.1944710Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:13:42.1944934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:13:42.1945038Z     return self.weight * hidden_states
2025-09-07T09:13:42.1945041Z 
2025-09-07T09:13:42.1945130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1945314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1945369Z     return mod(**inputs)
2025-09-07T09:13:42.1945597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1945660Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1945920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1945987Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1946198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1946267Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1946498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1946574Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1946800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1946906Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1947136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:13:42.1947205Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:13:42.1947208Z 
2025-09-07T09:13:42.1947298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1947481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1947536Z     return mod(**inputs)
2025-09-07T09:13:42.1947770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1947830Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1948057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1948120Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1948331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1948403Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1948631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1948743Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1948971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1949108Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1949342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:13:42.1949414Z     hidden_states = self.act(hidden_states)
2025-09-07T09:13:42.1949417Z 
2025-09-07T09:13:42.1949511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1949694Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1949752Z     return mod(**inputs)
2025-09-07T09:13:42.1949987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:13:42.1950052Z     decoder_outputs = self.decoder(
2025-09-07T09:13:42.1950285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:13:42.1950347Z     layer_outputs = layer_module(
2025-09-07T09:13:42.1950599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:13:42.1950672Z     return super().__call__(*args, **kwargs)
2025-09-07T09:13:42.1950900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:13:42.1950980Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:13:42.1951205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:13:42.1951307Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:13:42.1951570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:13:42.1951642Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:13:42.1951645Z 
2025-09-07T09:13:42.1951716Z cudagraph partition due to non gpu ops
2025-09-07T09:13:42.1951807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1951994Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1952052Z     return mod(**inputs)
2025-09-07T09:13:42.1952282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward
2025-09-07T09:13:42.1952395Z     sequence_output = sequence_output * (self.model_dim**-0.5)
2025-09-07T09:13:42.1952398Z 
2025-09-07T09:13:42.1952487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:13:42.1952673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:13:42.1952730Z     return mod(**inputs)
2025-09-07T09:13:42.1952965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward
2025-09-07T09:13:42.1953043Z     lm_logits = self.lm_head(sequence_output)
2025-09-07T09:13:42.1953048Z 
2025-09-07T09:13:58.8223124Z 
2025-09-07T09:13:58.9625423Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:13:59.1017825Z running benchmark:   3% 1/30 [00:00<00:04,  7.14it/s]
2025-09-07T09:13:59.2387536Z running benchmark:   7% 2/30 [00:00<00:03,  7.16it/s]
2025-09-07T09:13:59.3771622Z running benchmark:  10% 3/30 [00:00<00:03,  7.23it/s]
2025-09-07T09:13:59.5134818Z running benchmark:  13% 4/30 [00:00<00:03,  7.22it/s]
2025-09-07T09:13:59.6509226Z running benchmark:  17% 5/30 [00:00<00:03,  7.27it/s]
2025-09-07T09:13:59.8586455Z running benchmark:  20% 6/30 [00:00<00:03,  7.27it/s]
2025-09-07T09:13:59.9978921Z running benchmark:  23% 7/30 [00:01<00:03,  6.23it/s]
2025-09-07T09:14:00.1344918Z running benchmark:  27% 8/30 [00:01<00:03,  6.50it/s]
2025-09-07T09:14:00.2727785Z running benchmark:  30% 9/30 [00:01<00:03,  6.74it/s]
2025-09-07T09:14:00.4093340Z running benchmark:  33% 10/30 [00:01<00:02,  6.88it/s]
2025-09-07T09:14:00.5469045Z running benchmark:  37% 11/30 [00:01<00:02,  7.01it/s]
2025-09-07T09:14:00.6827206Z running benchmark:  40% 12/30 [00:01<00:02,  7.09it/s]
2025-09-07T09:14:00.8209900Z running benchmark:  43% 13/30 [00:01<00:02,  7.17it/s]
2025-09-07T09:14:00.9574852Z running benchmark:  47% 14/30 [00:01<00:02,  7.19it/s]
2025-09-07T09:14:01.0962326Z running benchmark:  50% 15/30 [00:02<00:02,  7.23it/s]
2025-09-07T09:14:01.2327512Z running benchmark:  53% 16/30 [00:02<00:01,  7.22it/s]
2025-09-07T09:14:01.3706190Z running benchmark:  57% 17/30 [00:02<00:01,  7.25it/s]
2025-09-07T09:14:01.5074614Z running benchmark:  60% 18/30 [00:02<00:01,  7.25it/s]
2025-09-07T09:14:01.6460313Z running benchmark:  63% 19/30 [00:02<00:01,  7.27it/s]
2025-09-07T09:14:01.7817156Z running benchmark:  67% 20/30 [00:02<00:01,  7.25it/s]
2025-09-07T09:14:01.9197787Z running benchmark:  70% 21/30 [00:02<00:01,  7.29it/s]
2025-09-07T09:14:02.0565892Z running benchmark:  73% 22/30 [00:03<00:01,  7.27it/s]
2025-09-07T09:14:02.1956017Z running benchmark:  77% 23/30 [00:03<00:00,  7.29it/s]
2025-09-07T09:14:02.3325938Z running benchmark:  80% 24/30 [00:03<00:00,  7.26it/s]
2025-09-07T09:14:02.4702066Z running benchmark:  83% 25/30 [00:03<00:00,  7.27it/s]
2025-09-07T09:14:02.6068203Z running benchmark:  87% 26/30 [00:03<00:00,  7.27it/s]
2025-09-07T09:14:02.7457363Z running benchmark:  90% 27/30 [00:03<00:00,  7.28it/s]
2025-09-07T09:14:02.8818209Z running benchmark:  93% 28/30 [00:03<00:00,  7.26it/s]
2025-09-07T09:14:03.0209710Z running benchmark:  97% 29/30 [00:04<00:00,  7.28it/s]
2025-09-07T09:14:03.0211210Z running benchmark: 100% 30/30 [00:04<00:00,  7.26it/s]
2025-09-07T09:14:03.0211486Z running benchmark: 100% 30/30 [00:04<00:00,  7.15it/s]
2025-09-07T09:14:03.1068007Z 1.932x
2025-09-07T09:14:03.1072518Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:14:05.4847762Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:14:05.4848692Z   import pynvml  # type: ignore[import]
2025-09-07T09:14:07.4986944Z 
2025-09-07T09:14:12.7731395Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:14:12.7732029Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:14:12.8144512Z cpu  eval  hf_T5_base                         
2025-09-07T09:14:29.6792799Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.6793123Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6793516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6793871Z     return mod(**inputs)
2025-09-07T09:14:29.6794305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6794701Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6795066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6795450Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6795802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6796171Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6796539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6796965Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6797333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6798072Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6798443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
2025-09-07T09:14:29.6798819Z     position_bias = position_bias + causal_mask
2025-09-07T09:14:29.6799113Z 
2025-09-07T09:14:29.6799223Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6799593Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6799967Z     return mod(**inputs)
2025-09-07T09:14:29.6800334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6800719Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6801104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6801490Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6801852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6802220Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6802604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6803203Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6803580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.6803991Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.6804393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.6804789Z     return self.weight * hidden_states
2025-09-07T09:14:29.6804926Z 
2025-09-07T09:14:29.6805040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6805488Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6805820Z     return mod(**inputs)
2025-09-07T09:14:29.6806173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6806546Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6806909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6807267Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6807616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6807976Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6808349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6808721Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6809079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6809460Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6809838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.6810214Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.6810343Z 
2025-09-07T09:14:29.6810465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6810821Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6811144Z     return mod(**inputs)
2025-09-07T09:14:29.6811492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6811876Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6812234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6812669Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6813012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6813433Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6813807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6814185Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6814555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6814932Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6815298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.6815661Z     key_states = self.k(current_states)
2025-09-07T09:14:29.6815789Z 
2025-09-07T09:14:29.6815889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6816240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6816578Z     return mod(**inputs)
2025-09-07T09:14:29.6817046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6817428Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6817867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6818231Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6818581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6818945Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6819363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6819760Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6820129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6820514Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6820875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.6821303Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.6821513Z 
2025-09-07T09:14:29.6821618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6821982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6822310Z     return mod(**inputs)
2025-09-07T09:14:29.6822654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6823034Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6823387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6823754Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6824102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6824459Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6824820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6825196Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6825564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6825930Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6826297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.6826828Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.6827248Z 
2025-09-07T09:14:29.6827345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6827740Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6828065Z     return mod(**inputs)
2025-09-07T09:14:29.6828406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6828767Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6829121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6829486Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6829826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6830191Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6830580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6830962Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6831358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6831741Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6832097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.6832465Z     value_states = self.v(current_states)
2025-09-07T09:14:29.6832602Z 
2025-09-07T09:14:29.6832698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6833057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6833379Z     return mod(**inputs)
2025-09-07T09:14:29.6833757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6834136Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6834521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6834909Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6835249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6835597Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6835966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6836340Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6836704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6837073Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6837437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.6837837Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.6838001Z 
2025-09-07T09:14:29.6838108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6838456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6838764Z     return mod(**inputs)
2025-09-07T09:14:29.6839105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6839468Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6839821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6840184Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6840558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6840908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6841265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6841679Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6842029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6842403Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6842761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.6843154Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.6843304Z 
2025-09-07T09:14:29.6843406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6843774Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6844089Z     return mod(**inputs)
2025-09-07T09:14:29.6844427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6844812Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6845200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6845564Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6845896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6846255Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6846609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6846978Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6847383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6847756Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6848124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.6848523Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.6848674Z 
2025-09-07T09:14:29.6848768Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6849112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6849429Z     return mod(**inputs)
2025-09-07T09:14:29.6849765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6850134Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6850482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6850848Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6851182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6851550Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6851901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6852275Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6852632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6853006Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6853362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.6853730Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.6853897Z 
2025-09-07T09:14:29.6853991Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6854340Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6854691Z     return mod(**inputs)
2025-09-07T09:14:29.6855021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.6855381Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.6855727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6856092Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6856428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6856777Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6857161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.6857600Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.6857969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.6858368Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.6858781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.6859159Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.6859298Z 
2025-09-07T09:14:29.6859395Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6859752Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6860076Z     return mod(**inputs)
2025-09-07T09:14:29.6860417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6860821Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6861180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6861557Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6861895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6862259Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6862630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6863028Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6863390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6863754Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6864143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.6864537Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.6864662Z 
2025-09-07T09:14:29.6864765Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6865109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6865430Z     return mod(**inputs)
2025-09-07T09:14:29.6865763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6866135Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6866487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6866849Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6867182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6867580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6867942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6868311Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6868717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6869100Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6869461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.6869827Z     key_states = self.k(current_states)
2025-09-07T09:14:29.6869961Z 
2025-09-07T09:14:29.6870060Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6870415Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6870752Z     return mod(**inputs)
2025-09-07T09:14:29.6871093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6871467Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6871811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6872212Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6872544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6872904Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6873262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6873622Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6873984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6874404Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6874772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.6875191Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.6875425Z 
2025-09-07T09:14:29.6875527Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6875871Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6876189Z     return mod(**inputs)
2025-09-07T09:14:29.6876521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6876891Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6877239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6877601Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6877938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6878292Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6878640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6879015Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6879375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6879748Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6880107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.6880553Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.6880761Z 
2025-09-07T09:14:29.6880857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6881288Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6881620Z     return mod(**inputs)
2025-09-07T09:14:29.6881952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6882359Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6882725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6883113Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6883456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6883816Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6884180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6884560Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6884927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6885293Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6887387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.6887894Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.6888106Z 
2025-09-07T09:14:29.6888202Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6888565Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6888880Z     return mod(**inputs)
2025-09-07T09:14:29.6889217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6889594Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6890006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6890376Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6890711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6891095Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6891459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6891834Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6892193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6892574Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6892934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.6893315Z     value_states = self.v(current_states)
2025-09-07T09:14:29.6893444Z 
2025-09-07T09:14:29.6893546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6893884Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6894205Z     return mod(**inputs)
2025-09-07T09:14:29.6894542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6894917Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6895281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6895635Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6895974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6896324Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6896694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6897118Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6897527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6897938Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6898315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.6898712Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.6898872Z 
2025-09-07T09:14:29.6898972Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6899332Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6899653Z     return mod(**inputs)
2025-09-07T09:14:29.6900001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6900383Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6900740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6901113Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6901490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6901863Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6902222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6902598Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6902996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6903513Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6903978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.6904390Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.6904554Z 
2025-09-07T09:14:29.6904655Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6905015Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6905342Z     return mod(**inputs)
2025-09-07T09:14:29.6905693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6906061Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6906428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6906796Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6907144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6907511Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6907876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6908255Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6908628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6909000Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6909358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.6909755Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.6909916Z 
2025-09-07T09:14:29.6910018Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6910373Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6910746Z     return mod(**inputs)
2025-09-07T09:14:29.6911083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6911500Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6911893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6912280Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6912617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6912987Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6913354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6913737Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6914114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6914481Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6914852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.6915236Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.6915409Z 
2025-09-07T09:14:29.6915494Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.6915734Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6916103Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6916428Z     return mod(**inputs)
2025-09-07T09:14:29.6916798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6917174Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6917564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6917945Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6918293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6918668Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6919042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.6919428Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.6919818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.6920217Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.6920617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.6920988Z     return self.weight * hidden_states
2025-09-07T09:14:29.6921126Z 
2025-09-07T09:14:29.6921224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6921577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6921900Z     return mod(**inputs)
2025-09-07T09:14:29.6922261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6922631Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6922990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6923356Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6923699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6924065Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6924425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.6924893Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.6925280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.6925731Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.6926134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.6926502Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.6926637Z 
2025-09-07T09:14:29.6926741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6927089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6927414Z     return mod(**inputs)
2025-09-07T09:14:29.6927743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6928111Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6928472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6928845Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6929219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6929581Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6929944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.6930328Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.6930705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.6931106Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.6931533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.6931917Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.6932060Z 
2025-09-07T09:14:29.6932155Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6932506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6932815Z     return mod(**inputs)
2025-09-07T09:14:29.6933157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6933533Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6933891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6934256Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6934585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6934948Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6935306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.6935697Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.6936066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.6936464Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.6936863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.6937232Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.6937361Z 
2025-09-07T09:14:29.6937442Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.6937724Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6938078Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6938443Z     return mod(**inputs)
2025-09-07T09:14:29.6938786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6939192Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6939540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6939912Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6940244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6940620Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6940973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6941353Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6941727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.6942129Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.6942520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.6942921Z     return self.weight * hidden_states
2025-09-07T09:14:29.6943052Z 
2025-09-07T09:14:29.6943145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6943494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6943810Z     return mod(**inputs)
2025-09-07T09:14:29.6944144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6944509Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6944895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6945275Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6945616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6945964Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6946325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6946689Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6947050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6947422Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6947778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.6948145Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.6948281Z 
2025-09-07T09:14:29.6948377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6948726Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6949040Z     return mod(**inputs)
2025-09-07T09:14:29.6949368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6949736Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6950092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6950456Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6950783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6951138Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6951498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6951920Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6952282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6952685Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6953048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.6953414Z     key_states = self.k(current_states)
2025-09-07T09:14:29.6953539Z 
2025-09-07T09:14:29.6953641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6953981Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6954288Z     return mod(**inputs)
2025-09-07T09:14:29.6954619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6954983Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6955335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6955690Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6956024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6956411Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6956773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6957139Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6957491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6957859Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6958217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.6958659Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.6958841Z 
2025-09-07T09:14:29.6958936Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6959280Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6959596Z     return mod(**inputs)
2025-09-07T09:14:29.6959928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6960293Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6960634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6960993Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6961329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6961685Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6962048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6962408Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6962769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6963139Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6963503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.6963944Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.6964156Z 
2025-09-07T09:14:29.6964250Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6964598Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6964912Z     return mod(**inputs)
2025-09-07T09:14:29.6965289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6965647Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6965995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6966397Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6966733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6967086Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6967444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6967809Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6968170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6968540Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6968891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.6969325Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.6969534Z 
2025-09-07T09:14:29.6969661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6970010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6970324Z     return mod(**inputs)
2025-09-07T09:14:29.6970647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6971006Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6971356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6971724Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6972096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6972453Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6972817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6973190Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6973550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6973913Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6974273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.6974638Z     value_states = self.v(current_states)
2025-09-07T09:14:29.6974766Z 
2025-09-07T09:14:29.6974866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6975211Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6975516Z     return mod(**inputs)
2025-09-07T09:14:29.6975844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6976207Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6976559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6976919Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6977251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6977639Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6978003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6978370Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6978770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6979134Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6979498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.6979934Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.6980096Z 
2025-09-07T09:14:29.6980196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6980531Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6980847Z     return mod(**inputs)
2025-09-07T09:14:29.6981177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6981540Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6981883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6982247Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6982577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6982931Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6983321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6983691Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6984054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6984425Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6984787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.6985212Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.6985377Z 
2025-09-07T09:14:29.6985476Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6985817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6986133Z     return mod(**inputs)
2025-09-07T09:14:29.6986463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6986823Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6987176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6987535Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6987864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6988218Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6988571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6988943Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6989316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6989692Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6990048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.6990443Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.6990606Z 
2025-09-07T09:14:29.6990700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6991046Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6991362Z     return mod(**inputs)
2025-09-07T09:14:29.6991689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6992099Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6992451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6992852Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6993181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6993535Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6993894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.6994261Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.6994622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.6994984Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.6995348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.6995718Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.6995844Z 
2025-09-07T09:14:29.6995944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.6996334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.6996644Z     return mod(**inputs)
2025-09-07T09:14:29.6996982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.6997347Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.6997697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.6998092Z     layer_outputs = layer_module(
2025-09-07T09:14:29.6998421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.6998808Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.6999171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.6999551Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.6999922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7000309Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7000687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7001051Z     return self.weight * hidden_states
2025-09-07T09:14:29.7001174Z 
2025-09-07T09:14:29.7001273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7001607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7001921Z     return mod(**inputs)
2025-09-07T09:14:29.7002273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7002634Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7002984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7021035Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7021442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7021839Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7022241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7022654Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7023054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7023589Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7023996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7024443Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7024577Z 
2025-09-07T09:14:29.7024692Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7025061Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7025390Z     return mod(**inputs)
2025-09-07T09:14:29.7025766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7026154Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7026519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7026884Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7027239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7027610Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7028040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7028432Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7028804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7029219Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7029624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7030007Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7030142Z 
2025-09-07T09:14:29.7030302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7030664Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7030997Z     return mod(**inputs)
2025-09-07T09:14:29.7031349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7031729Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7032088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7032463Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7032808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7033173Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7033544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7033923Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7034302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7034709Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7035111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7035477Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7035613Z 
2025-09-07T09:14:29.7035690Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7035923Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7036281Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7036602Z     return mod(**inputs)
2025-09-07T09:14:29.7036936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7037354Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7037715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7038081Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7038452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7038811Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7039179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7039553Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7039920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7040310Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7040704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7041073Z     return self.weight * hidden_states
2025-09-07T09:14:29.7041200Z 
2025-09-07T09:14:29.7041307Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7041661Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7042005Z     return mod(**inputs)
2025-09-07T09:14:29.7042345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7042711Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7043069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7043426Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7043760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7044154Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7044524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7044898Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7045257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7045632Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7045995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7046362Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7046489Z 
2025-09-07T09:14:29.7046592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7046932Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7047235Z     return mod(**inputs)
2025-09-07T09:14:29.7047562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7047918Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7048261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7048622Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7048944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7049290Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7049641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7050004Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7050354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7050787Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7051169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7051527Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7051689Z 
2025-09-07T09:14:29.7051798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7052181Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7052490Z     return mod(**inputs)
2025-09-07T09:14:29.7052819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7053173Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7053516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7053866Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7054198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7054626Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7054990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7055441Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7055807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7056181Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7056546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7056960Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7057149Z 
2025-09-07T09:14:29.7057245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7057690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7058018Z     return mod(**inputs)
2025-09-07T09:14:29.7058344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7058716Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7059068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7059432Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7059751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7060100Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7060470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7060849Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7061211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7061613Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7061974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7062419Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7062634Z 
2025-09-07T09:14:29.7062731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7063078Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7063396Z     return mod(**inputs)
2025-09-07T09:14:29.7063735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7064096Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7064442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7064846Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7065174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7065563Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7065923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7066280Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7066633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7067002Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7067357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7067800Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7068008Z 
2025-09-07T09:14:29.7068102Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7068440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7068755Z     return mod(**inputs)
2025-09-07T09:14:29.7069141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7069501Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7069854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7070220Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7070548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7070898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7071281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7071654Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7072009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7072374Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7072733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7073094Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7073226Z 
2025-09-07T09:14:29.7073317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7073651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7073959Z     return mod(**inputs)
2025-09-07T09:14:29.7074283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7074640Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7074982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7075345Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7075669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7076014Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7076366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7076724Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7077085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7077452Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7077841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7078290Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7078450Z 
2025-09-07T09:14:29.7079196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7079537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7079842Z     return mod(**inputs)
2025-09-07T09:14:29.7080167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7080529Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7080879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7081228Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7081552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7081898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7082254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7082623Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7083007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7083375Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7083725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7084107Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7084260Z 
2025-09-07T09:14:29.7084354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7084685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7085035Z     return mod(**inputs)
2025-09-07T09:14:29.7085364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7085718Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7086063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7086413Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7086736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7087103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7087454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7087838Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7088197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7088557Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7088933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7089323Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7089474Z 
2025-09-07T09:14:29.7089596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7089921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7090265Z     return mod(**inputs)
2025-09-07T09:14:29.7090603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7090969Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7091339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7091736Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7092061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7092430Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7092834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7093197Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7093568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7093959Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7094330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7094706Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7094834Z 
2025-09-07T09:14:29.7094908Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7095129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7095478Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7095791Z     return mod(**inputs)
2025-09-07T09:14:29.7096162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7096525Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7096870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7097227Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7097610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7097958Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7098349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7098732Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7099102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7099485Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7099854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7100213Z     return self.weight * hidden_states
2025-09-07T09:14:29.7100338Z 
2025-09-07T09:14:29.7100433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7100773Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7101079Z     return mod(**inputs)
2025-09-07T09:14:29.7101413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7101778Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7102125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7102478Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7102805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7103350Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7103705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7104074Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7104444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7104844Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7105243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7105672Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7105797Z 
2025-09-07T09:14:29.7105896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7106285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7106597Z     return mod(**inputs)
2025-09-07T09:14:29.7106927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7107292Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7107636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7107989Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7108317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7108667Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7109025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7109394Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7109807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7110208Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7110602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7110969Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7111100Z 
2025-09-07T09:14:29.7111199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7111543Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7111925Z     return mod(**inputs)
2025-09-07T09:14:29.7112257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7112616Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7112962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7113315Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7113640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7113986Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7114336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7114701Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7115066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7115455Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7115837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7116196Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7116324Z 
2025-09-07T09:14:29.7116398Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7116612Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7116949Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7117279Z     return mod(**inputs)
2025-09-07T09:14:29.7117600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7117953Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7118296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7118693Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7119014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7119415Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7119767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7120125Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7120480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7120868Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7121249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7121608Z     return self.weight * hidden_states
2025-09-07T09:14:29.7121733Z 
2025-09-07T09:14:29.7121836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7122173Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7122475Z     return mod(**inputs)
2025-09-07T09:14:29.7122839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7123205Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7123552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7123903Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7124231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7124574Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7124959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7125335Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7125684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7126051Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7126408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7126766Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7126890Z 
2025-09-07T09:14:29.7126982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7127319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7127624Z     return mod(**inputs)
2025-09-07T09:14:29.7127952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7128312Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7128656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7129010Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7129335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7129685Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7130031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7130396Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7130752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7131116Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7131473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7131873Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7132006Z 
2025-09-07T09:14:29.7132098Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7132436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7132779Z     return mod(**inputs)
2025-09-07T09:14:29.7133108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7133462Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7133810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7134164Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7134486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7134829Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7135186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7135549Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7135906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7136302Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7136653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7137062Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7137245Z 
2025-09-07T09:14:29.7137335Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7137713Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7138035Z     return mod(**inputs)
2025-09-07T09:14:29.7138398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7138768Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7139109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7139472Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7139791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7140139Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7140491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7140856Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7141215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7141575Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7141938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7142374Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7142581Z 
2025-09-07T09:14:29.7142678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7143017Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7143323Z     return mod(**inputs)
2025-09-07T09:14:29.7143648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7144009Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7144355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7144706Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7145033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7145417Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7145767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7146166Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7146513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7146880Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7147238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7147667Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7147865Z 
2025-09-07T09:14:29.7147961Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7148298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7148610Z     return mod(**inputs)
2025-09-07T09:14:29.7148943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7149299Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7149673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7150030Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7150356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7150704Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7151061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7151419Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7151809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7152198Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7152578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7152941Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7153077Z 
2025-09-07T09:14:29.7153172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7153514Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7153831Z     return mod(**inputs)
2025-09-07T09:14:29.7154161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7154525Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7154879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7155240Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7155578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7155939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7156305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7156665Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7157030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7157393Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7157745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7158153Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7158355Z 
2025-09-07T09:14:29.7158460Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7158805Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7159152Z     return mod(**inputs)
2025-09-07T09:14:29.7159479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7159839Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7160186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7160544Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7160866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7161220Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7161589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7161958Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7162310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7162667Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7163051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7163457Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7163613Z 
2025-09-07T09:14:29.7163708Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7164058Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7164364Z     return mod(**inputs)
2025-09-07T09:14:29.7164727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7165097Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7165448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7165821Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7166146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7166491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7166857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7167215Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7167570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7167933Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7168316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7168708Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7168859Z 
2025-09-07T09:14:29.7168955Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7169290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7169633Z     return mod(**inputs)
2025-09-07T09:14:29.7169961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7170319Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7170681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7171038Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7171372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7171778Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7172144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7172546Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7172906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7173278Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7173636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7174013Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7174135Z 
2025-09-07T09:14:29.7174226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7174567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7174884Z     return mod(**inputs)
2025-09-07T09:14:29.7175213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7175570Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7175922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7176317Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7176663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7177007Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7177359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7177766Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7178125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7178581Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7178769Z 
2025-09-07T09:14:29.7178844Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7178944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7179132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7179201Z     return mod(**inputs)
2025-09-07T09:14:29.7179431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7179504Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7179738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7179813Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7180034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7180103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7180334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7180421Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7180649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7180742Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7180967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7181041Z     return self.weight * hidden_states
2025-09-07T09:14:29.7181044Z 
2025-09-07T09:14:29.7181141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7181326Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7181393Z     return mod(**inputs)
2025-09-07T09:14:29.7181664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7181735Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7182003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7182090Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7182306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7182376Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7182607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7182713Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7182950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7183071Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7183302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7183381Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7183385Z 
2025-09-07T09:14:29.7183506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7183697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7183753Z     return mod(**inputs)
2025-09-07T09:14:29.7184010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7184074Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7184306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7184406Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7184624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7184701Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7184931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7185010Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7185261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7185367Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7185600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7185671Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7185674Z 
2025-09-07T09:14:29.7185765Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7185968Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7186029Z     return mod(**inputs)
2025-09-07T09:14:29.7186270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7186340Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7186579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7186643Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7186864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7186952Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7187190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7187313Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7187541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7187640Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7187909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7187981Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7187984Z 
2025-09-07T09:14:29.7188070Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7188160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7188346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7188415Z     return mod(**inputs)
2025-09-07T09:14:29.7188647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7188730Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7188966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7189040Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7189287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7189357Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7189596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7189675Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7189911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7190006Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7190265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7190352Z     return self.weight * hidden_states
2025-09-07T09:14:29.7190354Z 
2025-09-07T09:14:29.7190448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7190640Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7190697Z     return mod(**inputs)
2025-09-07T09:14:29.7190940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7191012Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7191243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7191310Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7191530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7191609Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7191837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7191908Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7192146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7192222Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7192453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7192522Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7192525Z 
2025-09-07T09:14:29.7192614Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7192801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7192857Z     return mod(**inputs)
2025-09-07T09:14:29.7193140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7193207Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7193443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7193551Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7193767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7193840Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7194068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7194163Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7194392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7194468Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7194700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7194771Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7194775Z 
2025-09-07T09:14:29.7194915Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7195100Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7195156Z     return mod(**inputs)
2025-09-07T09:14:29.7195392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7195474Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7195714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7195777Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7196049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7196126Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7196354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7196430Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7196657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7196731Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7196960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7197096Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7197100Z 
2025-09-07T09:14:29.7197195Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7197381Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7197445Z     return mod(**inputs)
2025-09-07T09:14:29.7197679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7197747Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7197988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7198073Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7198288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7198354Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7198588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7198658Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7198928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7199013Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7199243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7199428Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7199431Z 
2025-09-07T09:14:29.7199537Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7199726Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7199789Z     return mod(**inputs)
2025-09-07T09:14:29.7200021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7200091Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7200341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7200416Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7200627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7200695Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7200968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7201041Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7201274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7201343Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7201577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7201753Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7201758Z 
2025-09-07T09:14:29.7201850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7202042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7202102Z     return mod(**inputs)
2025-09-07T09:14:29.7202331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7202412Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7202641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7202708Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7202938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7203016Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7203394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7203466Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7203703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7203777Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7204026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7204100Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7204104Z 
2025-09-07T09:14:29.7204194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7204404Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7204460Z     return mod(**inputs)
2025-09-07T09:14:29.7204698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7204814Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7205056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7205180Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7205396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7205469Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7205699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7205786Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7206012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7206084Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7206318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7206428Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7206432Z 
2025-09-07T09:14:29.7206563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7206795Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7206855Z     return mod(**inputs)
2025-09-07T09:14:29.7207094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7207166Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7207453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7207523Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7207796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7207880Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7208118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7208201Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7208432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7208511Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7208747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7208848Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7208852Z 
2025-09-07T09:14:29.7208953Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7209148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7209234Z     return mod(**inputs)
2025-09-07T09:14:29.7209480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7209546Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7209787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7209850Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7210078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7210146Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7210377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7210446Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7210675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7210788Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7211015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7211149Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7211153Z 
2025-09-07T09:14:29.7211244Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7211429Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7211491Z     return mod(**inputs)
2025-09-07T09:14:29.7211724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7211792Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7212022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7212088Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7212314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7212382Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7212647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7212719Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7212951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7213023Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7213248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7213322Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7213326Z 
2025-09-07T09:14:29.7213399Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7213530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7213718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7213776Z     return mod(**inputs)
2025-09-07T09:14:29.7214014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7214078Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7214317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7214380Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7214597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7214671Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7214896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7214995Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7215222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7215316Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7215546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7215612Z     return self.weight * hidden_states
2025-09-07T09:14:29.7215615Z 
2025-09-07T09:14:29.7215711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7215895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7215959Z     return mod(**inputs)
2025-09-07T09:14:29.7216187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7216288Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7216523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7216587Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7216843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7216908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7217136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7217220Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7217447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7217645Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7217889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7217974Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7217978Z 
2025-09-07T09:14:29.7218067Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7218305Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7218375Z     return mod(**inputs)
2025-09-07T09:14:29.7218609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7218677Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7218906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7218968Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7219187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7219288Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7219526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7219602Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7219835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7219937Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7220164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7220242Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7220245Z 
2025-09-07T09:14:29.7220333Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7220531Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7220591Z     return mod(**inputs)
2025-09-07T09:14:29.7220832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7220903Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7221138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7221205Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7221419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7221484Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7221714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7221789Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7222035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7222170Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7222406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7222511Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7222514Z 
2025-09-07T09:14:29.7222585Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7222678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7222861Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7222923Z     return mod(**inputs)
2025-09-07T09:14:29.7223151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7223214Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7223449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7223513Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7223730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7223798Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7224060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7224137Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7224365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7224470Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7224699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7224768Z     return self.weight * hidden_states
2025-09-07T09:14:29.7224771Z 
2025-09-07T09:14:29.7224897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7225082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7225146Z     return mod(**inputs)
2025-09-07T09:14:29.7225385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7225452Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7225682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7225745Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7225962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7226032Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7226266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7226338Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7226581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7226656Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7226884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7226961Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7226964Z 
2025-09-07T09:14:29.7227052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7227256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7227320Z     return mod(**inputs)
2025-09-07T09:14:29.7227548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7227629Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7227895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7227963Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7228209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7228280Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7228519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7228591Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7228822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7228894Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7229138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7229210Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7229213Z 
2025-09-07T09:14:29.7229304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7229491Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7229552Z     return mod(**inputs)
2025-09-07T09:14:29.7229843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7229907Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7230140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7230213Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7230431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7230505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7230763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7230834Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7231070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7231145Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7231381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7231502Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7231506Z 
2025-09-07T09:14:29.7231600Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7231781Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7231837Z     return mod(**inputs)
2025-09-07T09:14:29.7232085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7232153Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7232391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7232468Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7232683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7232760Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7232987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7233081Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7233306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7233378Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7233642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7233789Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7233823Z 
2025-09-07T09:14:29.7233922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7234106Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7234171Z     return mod(**inputs)
2025-09-07T09:14:29.7234400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7234472Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7234710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7234771Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7234991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7235058Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7235286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7235394Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7235623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7235699Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7235928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7236069Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7236073Z 
2025-09-07T09:14:29.7236163Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7236382Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7236448Z     return mod(**inputs)
2025-09-07T09:14:29.7236680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7236752Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7236981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7237047Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7237277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7237347Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7237582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7237651Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7237880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7237957Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7238183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7238261Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7238264Z 
2025-09-07T09:14:29.7238351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7238539Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7238600Z     return mod(**inputs)
2025-09-07T09:14:29.7238832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7238900Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7239133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7239239Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7239460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7239574Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7239812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7239884Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7240124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7240194Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7240427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7240529Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7240534Z 
2025-09-07T09:14:29.7240624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7240815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7240875Z     return mod(**inputs)
2025-09-07T09:14:29.7241179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7241243Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7241474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7241543Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7241756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7241828Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7242094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7242174Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7242409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7242482Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7242720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7242825Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7242828Z 
2025-09-07T09:14:29.7242924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7243107Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7243164Z     return mod(**inputs)
2025-09-07T09:14:29.7243415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7243481Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7243717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7243782Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7243999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7244074Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7244301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7244376Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7244603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7244680Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7244917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7245044Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7245047Z 
2025-09-07T09:14:29.7245144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7245370Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7245442Z     return mod(**inputs)
2025-09-07T09:14:29.7245677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7245744Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7245992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7246058Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7246282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7246353Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7246591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7246674Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7246938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7247020Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7247250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7247327Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7247330Z 
2025-09-07T09:14:29.7247425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7247608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7247705Z     return mod(**inputs)
2025-09-07T09:14:29.7247942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7248024Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7248260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7248324Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7248547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7248615Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7248857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7248926Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7249153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7249285Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7249289Z 
2025-09-07T09:14:29.7249361Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7249462Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7249651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7249716Z     return mod(**inputs)
2025-09-07T09:14:29.7249946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7250010Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7250250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7250312Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7250529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7250632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7250861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7250984Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7251214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7251307Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7251550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7251622Z     return self.weight * hidden_states
2025-09-07T09:14:29.7251633Z 
2025-09-07T09:14:29.7251746Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7251933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7252001Z     return mod(**inputs)
2025-09-07T09:14:29.7252231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7252299Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7252563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7252630Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7252856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7252922Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7253157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7253241Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7253502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7253619Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7253847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7253928Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7253931Z 
2025-09-07T09:14:29.7254035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7254223Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7254283Z     return mod(**inputs)
2025-09-07T09:14:29.7254519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7254591Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7254828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7254902Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7255125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7255192Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7255431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7255512Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7255746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7255848Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7256078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7256155Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7256158Z 
2025-09-07T09:14:29.7256251Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7256479Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7256539Z     return mod(**inputs)
2025-09-07T09:14:29.7256773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7256870Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7257113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7257184Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7257395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7257470Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7257771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7257856Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7258089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7258201Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7258474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7258544Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7258547Z 
2025-09-07T09:14:29.7258629Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7258720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7258903Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7258964Z     return mod(**inputs)
2025-09-07T09:14:29.7259193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7259309Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7259547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7259609Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7259830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7259896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7260141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7260214Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7260442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7260545Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7260775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7260854Z     return self.weight * hidden_states
2025-09-07T09:14:29.7260857Z 
2025-09-07T09:14:29.7260952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7261143Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7261199Z     return mod(**inputs)
2025-09-07T09:14:29.7261428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7261504Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7261731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7261796Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7262011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7262133Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7262369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7262438Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7262709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7262782Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7263014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7263088Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7263092Z 
2025-09-07T09:14:29.7263181Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7263384Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7263444Z     return mod(**inputs)
2025-09-07T09:14:29.7263682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7263744Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7263976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7264085Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7275007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7275123Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7275406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7275490Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7275739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7275894Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7276143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7276214Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7276222Z 
2025-09-07T09:14:29.7276327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7276536Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7276599Z     return mod(**inputs)
2025-09-07T09:14:29.7276854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7276927Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7277176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7277247Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7277475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7277558Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7277798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7277875Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7278109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7278185Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7278420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7278544Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7278548Z 
2025-09-07T09:14:29.7278650Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7278855Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7278954Z     return mod(**inputs)
2025-09-07T09:14:29.7279199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7279300Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7279538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7279602Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7279822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7279898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7280131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7280211Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7280446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7280523Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7280761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7280947Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7280950Z 
2025-09-07T09:14:29.7281049Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7281243Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7281310Z     return mod(**inputs)
2025-09-07T09:14:29.7281555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7281622Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7281895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7281963Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7282187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7282262Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7282491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7282571Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7282803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7282880Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7283109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7283254Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7283260Z 
2025-09-07T09:14:29.7283351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7283538Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7283606Z     return mod(**inputs)
2025-09-07T09:14:29.7283838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7283908Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7284139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7284203Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7284422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7284490Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7284722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7284831Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7285058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7285170Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7285393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7285468Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7285472Z 
2025-09-07T09:14:29.7285565Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7285756Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7285814Z     return mod(**inputs)
2025-09-07T09:14:29.7286046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7286114Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7286346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7286416Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7286663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7286735Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7286971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7287041Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7287273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7287344Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7287609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7287717Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7287720Z 
2025-09-07T09:14:29.7287815Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7288005Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7288063Z     return mod(**inputs)
2025-09-07T09:14:29.7288298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7288360Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7288589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7288655Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7288871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7288947Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7289180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7289253Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7289487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7289558Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7289788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7289883Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7289886Z 
2025-09-07T09:14:29.7289981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7290165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7290258Z     return mod(**inputs)
2025-09-07T09:14:29.7290499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7290565Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7290841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7290905Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7291116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7291190Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7291419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7291496Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7291721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7291795Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7292028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7292126Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7292164Z 
2025-09-07T09:14:29.7292728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7292925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7292991Z     return mod(**inputs)
2025-09-07T09:14:29.7293241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7293311Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7293557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7293905Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7294134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7294208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7294440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7294515Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7294739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7294815Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7295040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7295116Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7295119Z 
2025-09-07T09:14:29.7295193Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7295286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7295480Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7295540Z     return mod(**inputs)
2025-09-07T09:14:29.7295774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7295839Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7296066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7296134Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7296346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7296420Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7296646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7296770Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7297004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7297122Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7297356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7297429Z     return self.weight * hidden_states
2025-09-07T09:14:29.7297433Z 
2025-09-07T09:14:29.7297601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7297788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7297848Z     return mod(**inputs)
2025-09-07T09:14:29.7298081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7298148Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7298400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7298465Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7298728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7298802Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7299037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7299126Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7299355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7299466Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7299729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7299802Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7299805Z 
2025-09-07T09:14:29.7299904Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7300101Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7300166Z     return mod(**inputs)
2025-09-07T09:14:29.7300398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7300464Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7300724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7300786Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7301007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7301077Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7301312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7301391Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7301630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7301737Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7301963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7302043Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7302046Z 
2025-09-07T09:14:29.7302138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7302323Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7302392Z     return mod(**inputs)
2025-09-07T09:14:29.7302659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7302729Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7302997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7303224Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7303448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7303517Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7303748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7303825Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7304053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7304165Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7304393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7304471Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7304474Z 
2025-09-07T09:14:29.7304610Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7304707Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7304894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7304953Z     return mod(**inputs)
2025-09-07T09:14:29.7305188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7305253Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7305542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7305607Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7305821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7305896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7306128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7306204Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7306428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7306526Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7306763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7306833Z     return self.weight * hidden_states
2025-09-07T09:14:29.7306836Z 
2025-09-07T09:14:29.7306927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7307113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7307171Z     return mod(**inputs)
2025-09-07T09:14:29.7307412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7307473Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7307703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7307767Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7307980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7308051Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7308278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7308400Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7308631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7308754Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7308991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7309062Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7309065Z 
2025-09-07T09:14:29.7309159Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7309340Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7309398Z     return mod(**inputs)
2025-09-07T09:14:29.7309632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7309697Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7309932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7309993Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7310206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7310313Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7310544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7310620Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7310845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7310924Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7311184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7311254Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7311257Z 
2025-09-07T09:14:29.7311350Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7311535Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7311596Z     return mod(**inputs)
2025-09-07T09:14:29.7311828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7311889Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7312122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7312187Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7312400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7312466Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7312695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7312768Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7312993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7313069Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7313293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7313417Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7313421Z 
2025-09-07T09:14:29.7313508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7313691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7313750Z     return mod(**inputs)
2025-09-07T09:14:29.7313980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7314079Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7314307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7314403Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7314619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7314684Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7314912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7314980Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7315208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7315278Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7315503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7315653Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7315658Z 
2025-09-07T09:14:29.7315792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7315980Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7316035Z     return mod(**inputs)
2025-09-07T09:14:29.7316263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7316326Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7316553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7316614Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7316858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7316925Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7317158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7317229Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7317455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7317523Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7317751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7317885Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7317890Z 
2025-09-07T09:14:29.7317983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7318188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7318246Z     return mod(**inputs)
2025-09-07T09:14:29.7318499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7318568Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7318812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7318875Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7319090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7319161Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7319389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7319460Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7319730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7319804Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7320067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7320137Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7320141Z 
2025-09-07T09:14:29.7320235Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7320425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7320482Z     return mod(**inputs)
2025-09-07T09:14:29.7320714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7320779Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7321023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7321086Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7321305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7321376Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7321645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7321717Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7321943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7322014Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7322237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7322368Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7322373Z 
2025-09-07T09:14:29.7322467Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7322652Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7322711Z     return mod(**inputs)
2025-09-07T09:14:29.7322943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7323006Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7323245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7323310Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7323523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7323588Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7323814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7323886Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7324110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7324185Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7324410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7324503Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7324506Z 
2025-09-07T09:14:29.7324593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7324774Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7324832Z     return mod(**inputs)
2025-09-07T09:14:29.7325060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7325164Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7325393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7325488Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7325704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7325770Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7325997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7326065Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7326293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7326362Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7326591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7326689Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7326692Z 
2025-09-07T09:14:29.7326780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7327001Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7327057Z     return mod(**inputs)
2025-09-07T09:14:29.7327287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7327350Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7327578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7327640Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7327882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7327957Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7328183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7328255Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7328482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7328554Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7328780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7328848Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7328851Z 
2025-09-07T09:14:29.7328940Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7329121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7329178Z     return mod(**inputs)
2025-09-07T09:14:29.7329409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7329473Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7329707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7329771Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7329985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7330049Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7330276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7330345Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7330570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7330735Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7330739Z 
2025-09-07T09:14:29.7330812Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7330903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7331122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7331179Z     return mod(**inputs)
2025-09-07T09:14:29.7331414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7331475Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7331706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7331765Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7331977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7332046Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7332271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7332355Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7332612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7332698Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7332927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7332992Z     return self.weight * hidden_states
2025-09-07T09:14:29.7332996Z 
2025-09-07T09:14:29.7333084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7333264Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7333354Z     return mod(**inputs)
2025-09-07T09:14:29.7333588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7333651Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7333886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7333945Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7334160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7334226Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7334457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7334539Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7334763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7334875Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7335103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7335174Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7335181Z 
2025-09-07T09:14:29.7335269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7335450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7335514Z     return mod(**inputs)
2025-09-07T09:14:29.7335742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7335810Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7336038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7336136Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7336352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7336418Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7336702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7336779Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7337005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7337112Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7337339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7337410Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7337413Z 
2025-09-07T09:14:29.7337551Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7337742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7337797Z     return mod(**inputs)
2025-09-07T09:14:29.7338027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7338128Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7338358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7338438Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7338661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7338729Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7338958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7339072Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7339305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7339406Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7339633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7339704Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7339707Z 
2025-09-07T09:14:29.7339776Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7339869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7340053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7340112Z     return mod(**inputs)
2025-09-07T09:14:29.7340341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7340407Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7340637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7340698Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7340913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7340979Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7341207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7341277Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7341502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7341599Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7341825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7341927Z     return self.weight * hidden_states
2025-09-07T09:14:29.7341933Z 
2025-09-07T09:14:29.7342021Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7342307Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7342365Z     return mod(**inputs)
2025-09-07T09:14:29.7342593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7342656Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7342882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7342943Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7343166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7343242Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7343476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7343548Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7343809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7343886Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7344110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7344179Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7344183Z 
2025-09-07T09:14:29.7344280Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7344476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7344536Z     return mod(**inputs)
2025-09-07T09:14:29.7344806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7344877Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7345107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7345176Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7345390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7345459Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7345687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7345757Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7345985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7346056Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7346282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7346351Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7346356Z 
2025-09-07T09:14:29.7346450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7346638Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7346696Z     return mod(**inputs)
2025-09-07T09:14:29.7346928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7346991Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7347227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7347294Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7347507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7347617Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7347843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7347945Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7348174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7348242Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7348468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7348592Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7348596Z 
2025-09-07T09:14:29.7348684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7348872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7348929Z     return mod(**inputs)
2025-09-07T09:14:29.7349163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7349226Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7349491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7349555Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7349767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7349836Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7350064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7350138Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7350398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7350471Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7350703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7350856Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7350859Z 
2025-09-07T09:14:29.7350950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7351133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7351193Z     return mod(**inputs)
2025-09-07T09:14:29.7351426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7351488Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7351724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7351784Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7351999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7352069Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7352295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7352370Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7352595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7352665Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7352891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7353028Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7353082Z 
2025-09-07T09:14:29.7353169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7353354Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7353453Z     return mod(**inputs)
2025-09-07T09:14:29.7353686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7353756Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7353984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7354045Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7354276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7354343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7354580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7354647Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7354875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7354986Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7355215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7355287Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7355290Z 
2025-09-07T09:14:29.7355378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7355561Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7355622Z     return mod(**inputs)
2025-09-07T09:14:29.7355887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7355954Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7356182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7356248Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7356460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7356528Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7356759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7356826Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7357052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7357122Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7357347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7357455Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7357458Z 
2025-09-07T09:14:29.7357546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7357730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7357787Z     return mod(**inputs)
2025-09-07T09:14:29.7358017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7358077Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7358305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7358367Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7358578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7358702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7358926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7359028Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7359257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7359325Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7359555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7359646Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7359649Z 
2025-09-07T09:14:29.7359736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7359923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7359980Z     return mod(**inputs)
2025-09-07T09:14:29.7360213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7360275Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7360546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7360608Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7360817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7360885Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7361112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7361180Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7361438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7361508Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7361736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7361827Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7361832Z 
2025-09-07T09:14:29.7361922Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7362106Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7362165Z     return mod(**inputs)
2025-09-07T09:14:29.7362398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7362461Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7362692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7362756Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7362970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7363035Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7363262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7363335Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7363562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7363634Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7363859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7363927Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7363930Z 
2025-09-07T09:14:29.7364006Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7364129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7364319Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7364407Z     return mod(**inputs)
2025-09-07T09:14:29.7364637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7364701Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7364928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7364989Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7365200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7365268Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7365496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7365579Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7365807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7365896Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7366164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7366233Z     return self.weight * hidden_states
2025-09-07T09:14:29.7366236Z 
2025-09-07T09:14:29.7366323Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7366509Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7366566Z     return mod(**inputs)
2025-09-07T09:14:29.7366802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7366902Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7367133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7367200Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7367413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7367484Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7367710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7367790Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7368015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7368125Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7368359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7368430Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7368433Z 
2025-09-07T09:14:29.7368526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7368713Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7368770Z     return mod(**inputs)
2025-09-07T09:14:29.7369002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7369062Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7369298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7369358Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7369575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7369677Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7369905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7369986Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7370249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7370353Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7370581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7370651Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7370655Z 
2025-09-07T09:14:29.7370748Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7370929Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7370991Z     return mod(**inputs)
2025-09-07T09:14:29.7371221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7371284Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7371551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7371614Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7371829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7371894Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7372125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7372200Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7372458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7372563Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7372789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7372860Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7372864Z 
2025-09-07T09:14:29.7372932Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7373020Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7373205Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7373260Z     return mod(**inputs)
2025-09-07T09:14:29.7373495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7373556Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7373789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7373853Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7374065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7374136Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7374370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7374442Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7374668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7374765Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7374995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7375058Z     return self.weight * hidden_states
2025-09-07T09:14:29.7375061Z 
2025-09-07T09:14:29.7375188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7375370Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7375425Z     return mod(**inputs)
2025-09-07T09:14:29.7375694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7375758Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7375992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7376052Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7376262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7376332Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7376556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7376627Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7376851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7376924Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7377184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7377253Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7377257Z 
2025-09-07T09:14:29.7377346Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7377599Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7377657Z     return mod(**inputs)
2025-09-07T09:14:29.7377886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7377998Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7378233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7378294Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7378510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7378576Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7378804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7378878Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7379103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7379175Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7379399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7379470Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7379474Z 
2025-09-07T09:14:29.7379562Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7379744Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7379807Z     return mod(**inputs)
2025-09-07T09:14:29.7380038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7380102Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7380329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7380389Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7380610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7380676Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7380944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7381013Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7381277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7381353Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7381583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7381708Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7381712Z 
2025-09-07T09:14:29.7381801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7381991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7382046Z     return mod(**inputs)
2025-09-07T09:14:29.7382277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7382348Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7382576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7382673Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7382887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7382955Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7383188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7383255Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7383488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7383592Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7383824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7383968Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7383973Z 
2025-09-07T09:14:29.7384063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7384248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7384302Z     return mod(**inputs)
2025-09-07T09:14:29.7384536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7384599Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7384829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7384893Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7385109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7385180Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7385407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7385483Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7385708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7385778Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7386010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7386146Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7386149Z 
2025-09-07T09:14:29.7386239Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7386465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7386523Z     return mod(**inputs)
2025-09-07T09:14:29.7386757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7386853Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7387091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7387154Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7387366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7387435Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7387660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7387730Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7387957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7388029Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7388256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7388355Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7388358Z 
2025-09-07T09:14:29.7388452Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7388634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7388696Z     return mod(**inputs)
2025-09-07T09:14:29.7388926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7388990Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7389259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7389323Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7389541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7389607Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7389836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7389912Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7390139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7390212Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7390437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7390541Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7390545Z 
2025-09-07T09:14:29.7390636Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7390818Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7390878Z     return mod(**inputs)
2025-09-07T09:14:29.7391111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7391176Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7391406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7391468Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7391685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7391750Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7391983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7392088Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7392316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7392419Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7392647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7392745Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7392748Z 
2025-09-07T09:14:29.7392837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7393023Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7393078Z     return mod(**inputs)
2025-09-07T09:14:29.7393306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7393372Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7393600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7393666Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7393912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7393982Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7394215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7394284Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7394513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7394582Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7394852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7394957Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7394960Z 
2025-09-07T09:14:29.7395055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7395254Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7395313Z     return mod(**inputs)
2025-09-07T09:14:29.7395554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7395622Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7395857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7395924Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7396144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7396219Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7396449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7396525Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7396754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7396829Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7397057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7397124Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7397127Z 
2025-09-07T09:14:29.7397225Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7397411Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7397470Z     return mod(**inputs)
2025-09-07T09:14:29.7397744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7397810Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7398080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7398145Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7398358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7398430Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7398658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7398730Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7398958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7399087Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7399090Z 
2025-09-07T09:14:29.7399164Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7399255Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7399494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7399553Z     return mod(**inputs)
2025-09-07T09:14:29.7399790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7399853Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7400081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7400145Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7400388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7400462Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7400688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7400774Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7401008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7401094Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7401324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7401390Z     return self.weight * hidden_states
2025-09-07T09:14:29.7401393Z 
2025-09-07T09:14:29.7401486Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7401673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7401733Z     return mod(**inputs)
2025-09-07T09:14:29.7401967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7402031Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7402266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7402328Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7402542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7402616Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7402842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7402924Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7403280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7403454Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7403681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7403804Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7403806Z 
2025-09-07T09:14:29.7403898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7404080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7404139Z     return mod(**inputs)
2025-09-07T09:14:29.7404368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7404431Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7404662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7404726Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7404943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7405008Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7405280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7405363Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7405587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7405694Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7405922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7405997Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7406000Z 
2025-09-07T09:14:29.7406143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7406331Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7406393Z     return mod(**inputs)
2025-09-07T09:14:29.7406622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7406692Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7406920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7406982Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7407200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7407264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7407492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7407592Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7407839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7407946Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7408177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7408247Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7408251Z 
2025-09-07T09:14:29.7408320Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7408411Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7408595Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7408650Z     return mod(**inputs)
2025-09-07T09:14:29.7408887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7408987Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7409224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7409319Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7409534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7409604Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7409829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7409900Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7410127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7410224Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7410454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7410523Z     return self.weight * hidden_states
2025-09-07T09:14:29.7410526Z 
2025-09-07T09:14:29.7410618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7410830Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7410890Z     return mod(**inputs)
2025-09-07T09:14:29.7411118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7411180Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7411413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7411474Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7411687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7411787Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7412019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7412093Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7412323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7412400Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7412626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7412696Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7412704Z 
2025-09-07T09:14:29.7412794Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7412976Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7413036Z     return mod(**inputs)
2025-09-07T09:14:29.7413271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7413338Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7413566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7413630Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7413843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7413909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7414140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7414209Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7414434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7414547Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7414772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7414845Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7414879Z 
2025-09-07T09:14:29.7414971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7415159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7415215Z     return mod(**inputs)
2025-09-07T09:14:29.7415443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7415511Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7415740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7415802Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7416017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7416084Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7416315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7416420Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7416653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7416725Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7416953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7417075Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7417078Z 
2025-09-07T09:14:29.7417167Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7417386Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7417444Z     return mod(**inputs)
2025-09-07T09:14:29.7417713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7417778Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7418007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7418074Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7418284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7418355Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7418582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7418649Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7418881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7418950Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7419181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7419332Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7419336Z 
2025-09-07T09:14:29.7419429Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7419614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7419670Z     return mod(**inputs)
2025-09-07T09:14:29.7419906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7419966Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7420203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7420310Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7420525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7420653Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7420880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7420952Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7421178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7421248Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7421481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7421621Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7421626Z 
2025-09-07T09:14:29.7421721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7421906Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7421969Z     return mod(**inputs)
2025-09-07T09:14:29.7422231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7422296Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7422532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7422595Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7422813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7422882Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7423145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7423221Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7423450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7423530Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7423756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7423824Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7423833Z 
2025-09-07T09:14:29.7423924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7424106Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7424168Z     return mod(**inputs)
2025-09-07T09:14:29.7424397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7424465Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7424695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7424756Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7424972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7425039Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7425271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7425339Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7425565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7425638Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7425864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7426004Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7426007Z 
2025-09-07T09:14:29.7426128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7426317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7426372Z     return mod(**inputs)
2025-09-07T09:14:29.7426602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7426668Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7426895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7426962Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7427173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7427240Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7427467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7427539Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7427809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7427881Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7428104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7428202Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7428205Z 
2025-09-07T09:14:29.7428292Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7428511Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7428571Z     return mod(**inputs)
2025-09-07T09:14:29.7428807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7428870Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7429099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7429164Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7429373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7429439Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7429665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7429734Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7429965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7430034Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7430264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7430357Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7430362Z 
2025-09-07T09:14:29.7430453Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7430634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7430688Z     return mod(**inputs)
2025-09-07T09:14:29.7430923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7430984Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7431216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7431342Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7431554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7431624Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7431888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7431965Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7432195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7432271Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7432501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7432569Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7432573Z 
2025-09-07T09:14:29.7432653Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7432745Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7432929Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7432995Z     return mod(**inputs)
2025-09-07T09:14:29.7433258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7433327Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7433559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7433628Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7433848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7433913Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7434179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7434262Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7434489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7434576Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7434803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7434874Z     return self.weight * hidden_states
2025-09-07T09:14:29.7434878Z 
2025-09-07T09:14:29.7434965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7435151Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7435207Z     return mod(**inputs)
2025-09-07T09:14:29.7435439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7435510Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7435737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7435803Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7436019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7436091Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7436318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7436394Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7436627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7436732Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7436965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7437075Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7437078Z 
2025-09-07T09:14:29.7437169Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7437402Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7437461Z     return mod(**inputs)
2025-09-07T09:14:29.7437693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7437756Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7437990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7438052Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7438261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7438333Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7438560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7438641Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7438910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7439014Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7439248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7439320Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7439323Z 
2025-09-07T09:14:29.7439415Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7439621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7439709Z     return mod(**inputs)
2025-09-07T09:14:29.7439949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7440012Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7440247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7440308Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7440525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7440594Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7440821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7440901Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7441128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7441238Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7441465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7441539Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7441542Z 
2025-09-07T09:14:29.7441623Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7441713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7441901Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7441957Z     return mod(**inputs)
2025-09-07T09:14:29.7442188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7442258Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7442492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7442621Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7442835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7442939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7443167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7443240Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7443473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7443567Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7443804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7443873Z     return self.weight * hidden_states
2025-09-07T09:14:29.7443876Z 
2025-09-07T09:14:29.7443971Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7444162Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7444216Z     return mod(**inputs)
2025-09-07T09:14:29.7444485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7444550Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7444787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7444847Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7445060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7445134Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7445394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7445470Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7445696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7445771Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7446007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7446074Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7446077Z 
2025-09-07T09:14:29.7446172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7446356Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7446415Z     return mod(**inputs)
2025-09-07T09:14:29.7446647Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7446710Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7446945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7447007Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7447222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7447292Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7447517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7447593Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7447817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7447892Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7448117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7448220Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7448223Z 
2025-09-07T09:14:29.7448315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7448497Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7448597Z     return mod(**inputs)
2025-09-07T09:14:29.7448829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7448892Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7449126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7449188Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7449405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7449472Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7449704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7449773Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7450001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7450113Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7450345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7450466Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7450470Z 
2025-09-07T09:14:29.7450558Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7450740Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7450801Z     return mod(**inputs)
2025-09-07T09:14:29.7451066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7451136Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7451371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7451439Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7451653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7451718Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7451947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7452015Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7452243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7452314Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7452540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7452682Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7452687Z 
2025-09-07T09:14:29.7452775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7452962Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7453019Z     return mod(**inputs)
2025-09-07T09:14:29.7453249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7453315Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7453545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7453611Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7453859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7453931Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7454158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7454258Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7454499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7454571Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7454805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7454947Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7454951Z 
2025-09-07T09:14:29.7455039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7455238Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7455298Z     return mod(**inputs)
2025-09-07T09:14:29.7455538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7455603Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7455867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7455934Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7456146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7456219Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7456448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7456524Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7456785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7456857Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7457094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7457166Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7457169Z 
2025-09-07T09:14:29.7457261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7457447Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7457582Z     return mod(**inputs)
2025-09-07T09:14:29.7457826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7457893Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7458132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7458197Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7458417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7458488Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7458716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7458796Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7459025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7459103Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7459327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7459427Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7459483Z 
2025-09-07T09:14:29.7459583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7459768Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7459866Z     return mod(**inputs)
2025-09-07T09:14:29.7460099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7460164Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7460402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7460466Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7460685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7460754Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7460989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7461062Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7461286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7461365Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7461639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7461740Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7461743Z 
2025-09-07T09:14:29.7461832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7462018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7462082Z     return mod(**inputs)
2025-09-07T09:14:29.7462369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7462440Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7462673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7462739Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7462960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7463026Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7463257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7463326Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7463559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7463632Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7463857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7463964Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7463968Z 
2025-09-07T09:14:29.7464055Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7464246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7464302Z     return mod(**inputs)
2025-09-07T09:14:29.7464532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7464599Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7464829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7464893Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7465103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7465208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7465434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7465536Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7465766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7465838Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7466072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7466141Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7466144Z 
2025-09-07T09:14:29.7466232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7466421Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7466481Z     return mod(**inputs)
2025-09-07T09:14:29.7466719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7466780Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7467044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7467119Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7467332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7467406Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7467634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7467712Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7467974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7468102Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7468105Z 
2025-09-07T09:14:29.7468185Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7468274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7468468Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7468525Z     return mod(**inputs)
2025-09-07T09:14:29.7468756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7468825Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7469056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7469124Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7469337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7469406Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7469636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7469716Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7469946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7470031Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7470261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7470330Z     return self.weight * hidden_states
2025-09-07T09:14:29.7470333Z 
2025-09-07T09:14:29.7470419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7470608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7470705Z     return mod(**inputs)
2025-09-07T09:14:29.7470941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7471003Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7471272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7471338Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7471548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7471622Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7471850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7471928Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7472163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7472268Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7472503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7472576Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7472579Z 
2025-09-07T09:14:29.7472711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7472905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7472964Z     return mod(**inputs)
2025-09-07T09:14:29.7473206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7473272Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7473511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7473610Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7473830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7473908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7474145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7474234Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7474462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7474576Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7474803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7474875Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7474878Z 
2025-09-07T09:14:29.7474981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7475170Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7475236Z     return mod(**inputs)
2025-09-07T09:14:29.7475468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7475535Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7475768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7475831Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7476052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7476121Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7476350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7476469Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7476695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7476806Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7477071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7477147Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7477150Z 
2025-09-07T09:14:29.7477224Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7477315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7477505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7477563Z     return mod(**inputs)
2025-09-07T09:14:29.7477798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:14:29.7477865Z     encoder_outputs = self.encoder(
2025-09-07T09:14:29.7478094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward
2025-09-07T09:14:29.7478196Z     hidden_states = self.final_layer_norm(hidden_states)
2025-09-07T09:14:29.7478460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7478533Z     return self.weight * hidden_states
2025-09-07T09:14:29.7478536Z 
2025-09-07T09:14:29.7478626Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7478813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7478869Z     return mod(**inputs)
2025-09-07T09:14:29.7479099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7479199Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7479438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7479505Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7479721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7479790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7480021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7480093Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7480324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7480397Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7480625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7480697Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7480701Z 
2025-09-07T09:14:29.7480788Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7480974Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7481034Z     return mod(**inputs)
2025-09-07T09:14:29.7481265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7481333Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7481561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7481630Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7481843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7481914Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7482179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7482253Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7482516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7482592Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7482827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7482947Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7482951Z 
2025-09-07T09:14:29.7483044Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7483229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7483289Z     return mod(**inputs)
2025-09-07T09:14:29.7483527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7483593Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7483825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7483937Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7484153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7484226Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7484455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7484527Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7484752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7484855Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7485087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7485231Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7485237Z 
2025-09-07T09:14:29.7485332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7485517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7485573Z     return mod(**inputs)
2025-09-07T09:14:29.7485808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7485870Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7486111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7486172Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7486391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7486459Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7486684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7486763Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7486989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7487067Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7487311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7492120Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7492132Z 
2025-09-07T09:14:29.7492273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7492492Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7492624Z     return mod(**inputs)
2025-09-07T09:14:29.7492892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7492999Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7493254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7493323Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7493551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7493626Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7493865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7493942Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7494170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7494251Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7494480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7494625Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7494629Z 
2025-09-07T09:14:29.7494733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7494935Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7495004Z     return mod(**inputs)
2025-09-07T09:14:29.7495243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7495319Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7495584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7495656Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7495885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7495959Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7496196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7496268Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7496496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7496571Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7496797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7496897Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7496903Z 
2025-09-07T09:14:29.7496997Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7497197Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7497260Z     return mod(**inputs)
2025-09-07T09:14:29.7497562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7497636Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7497866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7497933Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7498154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7498224Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7498458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7498569Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7498804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7498911Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7499147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7499246Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7499250Z 
2025-09-07T09:14:29.7499343Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7499542Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7499602Z     return mod(**inputs)
2025-09-07T09:14:29.7499841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7499906Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7500136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7500204Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7500453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7500526Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7500758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7500830Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7501060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7501132Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7501398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7501470Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7501473Z 
2025-09-07T09:14:29.7501550Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7501645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7501831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7501893Z     return mod(**inputs)
2025-09-07T09:14:29.7502122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7502191Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7502422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7502485Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7502702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7502769Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7502999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7503251Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7503482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7503574Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7503799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7503873Z     return self.weight * hidden_states
2025-09-07T09:14:29.7503876Z 
2025-09-07T09:14:29.7503966Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7504156Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7504292Z     return mod(**inputs)
2025-09-07T09:14:29.7504526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7504645Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7504879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7504948Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7505159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7505226Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7505467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7505549Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7505783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7505897Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7506127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7506254Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7506257Z 
2025-09-07T09:14:29.7506349Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7506541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7506599Z     return mod(**inputs)
2025-09-07T09:14:29.7506834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7506897Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7507172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7507245Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7507460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7507536Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7507765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7507842Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7508077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7508178Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7508414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7508495Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7508499Z 
2025-09-07T09:14:29.7508596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7508788Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7508848Z     return mod(**inputs)
2025-09-07T09:14:29.7509086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7509150Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7509382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7509446Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7509658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7509732Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7509958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7510078Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7510303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7510450Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7510678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7510747Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7510750Z 
2025-09-07T09:14:29.7510844Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7511026Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7511086Z     return mod(**inputs)
2025-09-07T09:14:29.7511319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7511388Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7511626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7511689Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7511944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7512012Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7512242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7512322Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7512552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7512658Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7512918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7512991Z     return self.weight * hidden_states
2025-09-07T09:14:29.7513002Z 
2025-09-07T09:14:29.7513095Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7513281Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7513348Z     return mod(**inputs)
2025-09-07T09:14:29.7513580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7513653Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7513883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7513946Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7514166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7514234Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7514470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7514545Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7514781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7514860Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7515088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7515163Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7515166Z 
2025-09-07T09:14:29.7515256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7515444Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7515506Z     return mod(**inputs)
2025-09-07T09:14:29.7515738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7516007Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7516243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7516353Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7516566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7516635Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7516867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7516937Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7517170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7517241Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7517471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7517543Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7517548Z 
2025-09-07T09:14:29.7517638Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7517860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7517920Z     return mod(**inputs)
2025-09-07T09:14:29.7518159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7518223Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7518455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7518522Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7518778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7518855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7519085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7519157Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7519389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7519462Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7519694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7519821Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7519825Z 
2025-09-07T09:14:29.7519921Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7520109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7520169Z     return mod(**inputs)
2025-09-07T09:14:29.7520405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7520470Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7520711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7520775Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7520987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7521055Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7521280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7521351Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7521576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7521681Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7521909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7522093Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7522097Z 
2025-09-07T09:14:29.7522188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7522371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7522432Z     return mod(**inputs)
2025-09-07T09:14:29.7522662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7522724Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7522958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7523020Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7523233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7523300Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7523558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7523630Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7523856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7523927Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7524153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7524221Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7524288Z 
2025-09-07T09:14:29.7524377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7524557Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7524620Z     return mod(**inputs)
2025-09-07T09:14:29.7524854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7524915Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7525148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7525208Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7525420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7525487Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7525714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7525785Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7526010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7526082Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7526307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7526402Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7526405Z 
2025-09-07T09:14:29.7526495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7526678Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7526739Z     return mod(**inputs)
2025-09-07T09:14:29.7526970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7527067Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7527300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7527395Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7527611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7527676Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7527901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7527974Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7528199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7528272Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7528501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7528597Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7528600Z 
2025-09-07T09:14:29.7528688Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7528909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7528975Z     return mod(**inputs)
2025-09-07T09:14:29.7529204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7529270Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7529497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7529560Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7529775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7529873Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7530108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7530180Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7530427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7530498Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7530724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7530822Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7530825Z 
2025-09-07T09:14:29.7530914Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7531099Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7531157Z     return mod(**inputs)
2025-09-07T09:14:29.7531387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7531456Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7531686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7531751Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7531961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7532029Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7532260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7532328Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7532559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7532682Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7532910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7533012Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7533015Z 
2025-09-07T09:14:29.7533087Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7533180Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7533361Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7533422Z     return mod(**inputs)
2025-09-07T09:14:29.7533654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7533717Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7533951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7534013Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7534228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7534293Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7534559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7534635Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7534861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7534963Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7535188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7535256Z     return self.weight * hidden_states
2025-09-07T09:14:29.7535259Z 
2025-09-07T09:14:29.7535383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7535566Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7535627Z     return mod(**inputs)
2025-09-07T09:14:29.7535859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7535925Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7536155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7536217Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7536435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7536499Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7536732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7536805Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7537114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7537197Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7537445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7537553Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7537556Z 
2025-09-07T09:14:29.7537653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7537850Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7537906Z     return mod(**inputs)
2025-09-07T09:14:29.7538143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7538213Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7538492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7538559Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7538779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7538884Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7539122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7539194Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7539425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7539500Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7539726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7539802Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7539805Z 
2025-09-07T09:14:29.7539901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7540092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7540151Z     return mod(**inputs)
2025-09-07T09:14:29.7540423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7543072Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7543346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7543420Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7544998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7545084Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7545336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7545415Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7545646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7545733Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7545984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7546110Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7546114Z 
2025-09-07T09:14:29.7546223Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7546427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7546495Z     return mod(**inputs)
2025-09-07T09:14:29.7546740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7546812Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7547050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7547116Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7547336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7547417Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7547650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7547728Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7547956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7548039Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7548342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7548494Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7548532Z 
2025-09-07T09:14:29.7548635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7548831Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7548895Z     return mod(**inputs)
2025-09-07T09:14:29.7549131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7549200Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7549438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7549504Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7549722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7549801Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7550027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7550105Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7550330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7550483Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7550718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7550790Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7550793Z 
2025-09-07T09:14:29.7550931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7551122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7551184Z     return mod(**inputs)
2025-09-07T09:14:29.7551417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7551485Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7551721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7551783Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7552000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7552068Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7552293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7552372Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7552600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7552680Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7552903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7553008Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7553012Z 
2025-09-07T09:14:29.7553108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7553296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7553356Z     return mod(**inputs)
2025-09-07T09:14:29.7553588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7553652Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7553886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7553997Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7554218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7554322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7554555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7554624Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7554855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7554935Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7555162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7555260Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7555265Z 
2025-09-07T09:14:29.7555354Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7555540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7555607Z     return mod(**inputs)
2025-09-07T09:14:29.7555839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7555906Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7556208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7556276Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7556491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7556557Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7556825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7556900Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7557130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7557205Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7557433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7557535Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7557539Z 
2025-09-07T09:14:29.7557629Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7557821Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7557879Z     return mod(**inputs)
2025-09-07T09:14:29.7558109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7558175Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7558402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7558469Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7558682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7558749Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7558980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7559055Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7559294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7559374Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7559604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7559721Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7559724Z 
2025-09-07T09:14:29.7559799Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7559932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7560124Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7560188Z     return mod(**inputs)
2025-09-07T09:14:29.7560424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7560491Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7560729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7560794Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7561014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7561085Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7561321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7561406Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7561631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7561762Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7561993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7562068Z     return self.weight * hidden_states
2025-09-07T09:14:29.7562072Z 
2025-09-07T09:14:29.7562166Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7562391Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7562456Z     return mod(**inputs)
2025-09-07T09:14:29.7562689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7562758Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7562988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7563050Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7563269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7563338Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7563570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7563650Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7563884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7563999Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7564226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7564303Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7564307Z 
2025-09-07T09:14:29.7564399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7564590Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7564646Z     return mod(**inputs)
2025-09-07T09:14:29.7564874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7564939Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7565169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7565275Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7565490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7565557Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7565824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7565903Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7566134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7566243Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7566472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7566545Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7566549Z 
2025-09-07T09:14:29.7566641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7566829Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7566887Z     return mod(**inputs)
2025-09-07T09:14:29.7567126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7567191Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7567455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7567520Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7567731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7567799Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7568057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7568137Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7568363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7568464Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7568695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7568766Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7568771Z 
2025-09-07T09:14:29.7568846Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7568940Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7569130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7569189Z     return mod(**inputs)
2025-09-07T09:14:29.7569418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7569487Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7569717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7569780Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7569996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7570062Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7570293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7570365Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7570594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7570696Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7570922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7571032Z     return self.weight * hidden_states
2025-09-07T09:14:29.7571035Z 
2025-09-07T09:14:29.7571123Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7571346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7571405Z     return mod(**inputs)
2025-09-07T09:14:29.7571637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7571706Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7571933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7572001Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7572216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7572287Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7572519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7572591Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7572819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7572892Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7573168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7573238Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7573241Z 
2025-09-07T09:14:29.7573334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7573560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7573618Z     return mod(**inputs)
2025-09-07T09:14:29.7573855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7573921Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7574154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7574222Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7574434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7574507Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7574736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7574806Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7575035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7575107Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7575340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7575412Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7575416Z 
2025-09-07T09:14:29.7575507Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7575691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7575750Z     return mod(**inputs)
2025-09-07T09:14:29.7575987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7576051Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7576283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7576346Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7576602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7576674Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7576950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7577030Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7577259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7577336Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7577629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7577756Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7577762Z 
2025-09-07T09:14:29.7577861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7578045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7578109Z     return mod(**inputs)
2025-09-07T09:14:29.7578340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7578407Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7578643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7578748Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7578966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7579032Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7579292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7579367Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7579594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7579667Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7579894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7580044Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7580047Z 
2025-09-07T09:14:29.7580138Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7580320Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7580382Z     return mod(**inputs)
2025-09-07T09:14:29.7580613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7580679Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7580909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7580970Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7581185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7581255Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7581487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7581561Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7581790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7581860Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7582085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7582193Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7582196Z 
2025-09-07T09:14:29.7582285Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7582470Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7582560Z     return mod(**inputs)
2025-09-07T09:14:29.7582789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7582854Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7583087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7583151Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7583363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7583429Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7583656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7583725Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7583952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7584022Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7584249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7584389Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7584392Z 
2025-09-07T09:14:29.7584484Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7584670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7584761Z     return mod(**inputs)
2025-09-07T09:14:29.7584997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7585062Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7585291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7585355Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7585566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7585638Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7585865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7585935Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7586166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7586237Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7586468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7586561Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7586565Z 
2025-09-07T09:14:29.7586658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7586840Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7586898Z     return mod(**inputs)
2025-09-07T09:14:29.7587132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7587197Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7587433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7587496Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7587712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7587821Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7588048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7588156Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7588383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7588455Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7588690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7588783Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7588786Z 
2025-09-07T09:14:29.7588883Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7589069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7589132Z     return mod(**inputs)
2025-09-07T09:14:29.7589362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7589426Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7589659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7589719Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7590173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7590242Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7590476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7590586Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7590817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7590892Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7591118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7591188Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7591191Z 
2025-09-07T09:14:29.7591282Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7591467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7591527Z     return mod(**inputs)
2025-09-07T09:14:29.7591755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7591820Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7592049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7592111Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7592322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7592387Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7592617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7592685Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7592911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7593038Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7593041Z 
2025-09-07T09:14:29.7593114Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7593205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7593389Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7593484Z     return mod(**inputs)
2025-09-07T09:14:29.7593713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7593809Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7594040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7594103Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7594322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7594387Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7594613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7594685Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7594908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7595007Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7595231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7595301Z     return self.weight * hidden_states
2025-09-07T09:14:29.7595307Z 
2025-09-07T09:14:29.7595398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7595615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7595675Z     return mod(**inputs)
2025-09-07T09:14:29.7595905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7595969Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7596231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7596298Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7596511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7596578Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7596809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7596877Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7597104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7597178Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7597406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7597480Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7597484Z 
2025-09-07T09:14:29.7597574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7597766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7597826Z     return mod(**inputs)
2025-09-07T09:14:29.7598055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7598130Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7598361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7598427Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7598637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7598704Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7598934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7599059Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7599289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7599362Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7599625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7599698Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7599702Z 
2025-09-07T09:14:29.7599792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7599979Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7600035Z     return mod(**inputs)
2025-09-07T09:14:29.7600275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7600339Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7600570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7600638Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7600849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7600919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7601145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7601251Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7601490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7601566Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7601828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7601950Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7601954Z 
2025-09-07T09:14:29.7602044Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7602229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7602289Z     return mod(**inputs)
2025-09-07T09:14:29.7602524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7602587Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7602822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7602885Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7603272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7603359Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7603590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7603662Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7603890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7603963Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7604193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7604338Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7604341Z 
2025-09-07T09:14:29.7604439Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7604623Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7604685Z     return mod(**inputs)
2025-09-07T09:14:29.7604916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7605060Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7605292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7605406Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7605625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7605696Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7605926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7605999Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7606228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7606304Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7606531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7606602Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7606607Z 
2025-09-07T09:14:29.7606704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7606896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7606957Z     return mod(**inputs)
2025-09-07T09:14:29.7607244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7607311Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7607545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7607656Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7607879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7607951Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7608183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7608254Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7608480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7608557Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7608783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7608888Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7608892Z 
2025-09-07T09:14:29.7608984Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7609168Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7609236Z     return mod(**inputs)
2025-09-07T09:14:29.7609466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7609535Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7609765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7609833Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7610046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7610113Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7610341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7610412Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7610640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7610750Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7610978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7611111Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7611114Z 
2025-09-07T09:14:29.7611205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7611395Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7611454Z     return mod(**inputs)
2025-09-07T09:14:29.7611685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7611753Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7611981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7612049Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7612263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7612333Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7612559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7612629Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7612893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7612967Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7613232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7613324Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7613329Z 
2025-09-07T09:14:29.7613419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7613605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7613663Z     return mod(**inputs)
2025-09-07T09:14:29.7613896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7613958Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7614193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7614253Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7614464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7614538Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7614763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7614841Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7615064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7615137Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7615367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7615435Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7615442Z 
2025-09-07T09:14:29.7615518Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7615608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7615794Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7615857Z     return mod(**inputs)
2025-09-07T09:14:29.7616085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7616189Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7616421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7616552Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7616763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7616834Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7617072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7617154Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7617386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7617474Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7617743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7617821Z     return self.weight * hidden_states
2025-09-07T09:14:29.7617824Z 
2025-09-07T09:14:29.7617914Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7618109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7618165Z     return mod(**inputs)
2025-09-07T09:14:29.7618439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7618507Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7618740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7618809Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7619069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7619142Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7619372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7619453Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7619682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7619790Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7620018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7620092Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7620095Z 
2025-09-07T09:14:29.7620188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7620376Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7620434Z     return mod(**inputs)
2025-09-07T09:14:29.7620670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7620735Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7620965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7621030Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7621241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7621313Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7621540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7621620Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7621848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7621987Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7622219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7622325Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7622329Z 
2025-09-07T09:14:29.7622422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7622607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7622664Z     return mod(**inputs)
2025-09-07T09:14:29.7622894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7622956Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7623189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7623252Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7623467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7623539Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7623766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7623849Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7624113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7624221Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7624447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7624554Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7624557Z 
2025-09-07T09:14:29.7624637Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7624724Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7624915Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7624974Z     return mod(**inputs)
2025-09-07T09:14:29.7625202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7625270Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7625500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7625563Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7625773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7625847Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7626075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7626149Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7626382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7626478Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7626706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7626777Z     return self.weight * hidden_states
2025-09-07T09:14:29.7626780Z 
2025-09-07T09:14:29.7626868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7627055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7627110Z     return mod(**inputs)
2025-09-07T09:14:29.7627344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7627442Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7627673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7627775Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7627988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7628057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7628287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7628363Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7628588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7628662Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7628892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7628963Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7628967Z 
2025-09-07T09:14:29.7629056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7629239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7629294Z     return mod(**inputs)
2025-09-07T09:14:29.7629562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7629626Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7629860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7629922Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7630169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7630245Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7630472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7630552Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7630777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7630848Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7631077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7631146Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7631150Z 
2025-09-07T09:14:29.7631243Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7631428Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7631488Z     return mod(**inputs)
2025-09-07T09:14:29.7631718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7631781Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7632019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7632080Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7632292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7632358Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7632586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7632659Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7632883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7633000Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7633226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7633390Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7633393Z 
2025-09-07T09:14:29.7633482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7633668Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7633731Z     return mod(**inputs)
2025-09-07T09:14:29.7633960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7634024Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7634255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7634316Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7634531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7634596Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7634831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7634898Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7635162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7635233Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7635460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7635648Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7635651Z 
2025-09-07T09:14:29.7635739Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7635929Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7635984Z     return mod(**inputs)
2025-09-07T09:14:29.7636216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7636283Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7636514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7636578Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7636792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7636856Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7637089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7637160Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7637390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7637458Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7637691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7637764Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7637767Z 
2025-09-07T09:14:29.7637854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7638041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7638096Z     return mod(**inputs)
2025-09-07T09:14:29.7638328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7638392Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7638621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7638720Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7638932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7639033Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7639262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7639333Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7639565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7639634Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7639866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7639962Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7639966Z 
2025-09-07T09:14:29.7640060Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7640242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7640303Z     return mod(**inputs)
2025-09-07T09:14:29.7640536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7640600Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7640894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7640958Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7641171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7641284Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7641515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7641593Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7641819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7641897Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7642123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7642220Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7642224Z 
2025-09-07T09:14:29.7642318Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7642500Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7642560Z     return mod(**inputs)
2025-09-07T09:14:29.7642790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7642852Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7643088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7643152Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7643374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7643440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7643669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7643745Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7643973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7644048Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7644314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7644409Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7644412Z 
2025-09-07T09:14:29.7644533Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7644718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7644781Z     return mod(**inputs)
2025-09-07T09:14:29.7645012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7645083Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7645312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7645375Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7645592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7645658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7645893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7645965Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7646192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7646266Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7646523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7646600Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7646603Z 
2025-09-07T09:14:29.7646672Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7646797Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7646982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7647041Z     return mod(**inputs)
2025-09-07T09:14:29.7647273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7647337Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7647573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7647635Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7647852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7647924Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7648151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7648227Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7648452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7648550Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7648776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7648844Z     return self.weight * hidden_states
2025-09-07T09:14:29.7648847Z 
2025-09-07T09:14:29.7648938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7649121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7649179Z     return mod(**inputs)
2025-09-07T09:14:29.7649409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7649470Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7649703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7649797Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7650012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7650112Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7650338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7650411Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7650638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7650712Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7650938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7651010Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7651015Z 
2025-09-07T09:14:29.7651104Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7651285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7651345Z     return mod(**inputs)
2025-09-07T09:14:29.7651577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7651642Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7651902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7651966Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7652179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7652244Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7652508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7652580Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7652808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7652883Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7653112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7653181Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7653186Z 
2025-09-07T09:14:29.7653275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7653458Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7653515Z     return mod(**inputs)
2025-09-07T09:14:29.7653748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7653814Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7654043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7654109Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7654322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7654387Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7654622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7654690Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7654924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7654995Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7655224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7655380Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7655383Z 
2025-09-07T09:14:29.7655471Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7655697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7655754Z     return mod(**inputs)
2025-09-07T09:14:29.7655993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7656055Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7656285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7656359Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7656572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7656647Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7656870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7656942Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7657177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7657249Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7657569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7657716Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7657720Z 
2025-09-07T09:14:29.7657817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7658043Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7658105Z     return mod(**inputs)
2025-09-07T09:14:29.7658342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7658405Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7658645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7658708Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7658920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7658997Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7659227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7659304Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7659533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7659609Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7659846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7659919Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7659922Z 
2025-09-07T09:14:29.7660014Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7660196Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7660260Z     return mod(**inputs)
2025-09-07T09:14:29.7660493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7660556Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7660792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7660856Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7661142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7661208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7661501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7661580Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7661810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7661887Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7662115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7662210Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7662221Z 
2025-09-07T09:14:29.7662311Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7662496Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7662558Z     return mod(**inputs)
2025-09-07T09:14:29.7662789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7662861Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7663090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7663188Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7663409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7663479Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7663742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7663815Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7664046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7664125Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7664352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7664453Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7664456Z 
2025-09-07T09:14:29.7664547Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7664738Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7664796Z     return mod(**inputs)
2025-09-07T09:14:29.7665027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7665099Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7665328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7665397Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7665608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7665677Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7665911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7665984Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7666219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7666291Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7666523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7666622Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7666663Z 
2025-09-07T09:14:29.7666755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7666944Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7667040Z     return mod(**inputs)
2025-09-07T09:14:29.7667280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7667346Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7667577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7667648Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7667865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7667939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7668163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7668237Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7668470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7668544Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7668775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7668877Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7668880Z 
2025-09-07T09:14:29.7668979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7669165Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7669225Z     return mod(**inputs)
2025-09-07T09:14:29.7669491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7669557Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7669797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7669861Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7670071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7670141Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7670368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7670449Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7670675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:14:29.7670802Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7670807Z 
2025-09-07T09:14:29.7670888Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7670980Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7671172Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7671233Z     return mod(**inputs)
2025-09-07T09:14:29.7671469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7671542Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7671773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7671840Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7672052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7672129Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7672358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7672474Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7672710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7672833Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7673066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7673139Z     return self.weight * hidden_states
2025-09-07T09:14:29.7673142Z 
2025-09-07T09:14:29.7673232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7673427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7673491Z     return mod(**inputs)
2025-09-07T09:14:29.7673727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7673795Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7674032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7674095Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7674311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7674386Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7674643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7674729Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7674961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7675103Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7675344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7675418Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7675422Z 
2025-09-07T09:14:29.7675518Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7675705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7675764Z     return mod(**inputs)
2025-09-07T09:14:29.7676005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7676071Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7676306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7676369Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7676589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7676658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7676883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7676968Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7677198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7677310Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7677537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7677614Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7677617Z 
2025-09-07T09:14:29.7677714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7677899Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7677996Z     return mod(**inputs)
2025-09-07T09:14:29.7678229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7678334Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7678567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7678627Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7678850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7678919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7679152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7679228Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7679457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7679565Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7679791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7679868Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7679871Z 
2025-09-07T09:14:29.7679941Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7680061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7680255Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7680313Z     return mod(**inputs)
2025-09-07T09:14:29.7680554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7680650Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7680892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7680959Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7681171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7681240Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7681464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7681542Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7681768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7681864Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7682094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7682161Z     return self.weight * hidden_states
2025-09-07T09:14:29.7682165Z 
2025-09-07T09:14:29.7682263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7682445Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7682505Z     return mod(**inputs)
2025-09-07T09:14:29.7682739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7682807Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7683043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7683106Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7683320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7683388Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7683617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7683744Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7683985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7684102Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7684336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7684408Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7684412Z 
2025-09-07T09:14:29.7684517Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7684711Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7684776Z     return mod(**inputs)
2025-09-07T09:14:29.7685014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7685084Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7685319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7685384Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7685606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7685679Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7685942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7686017Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7686243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7686350Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7686581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7686656Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7686660Z 
2025-09-07T09:14:29.7686752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7686942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7687004Z     return mod(**inputs)
2025-09-07T09:14:29.7687238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7687307Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7687535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7687600Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7687815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7687887Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7688119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7688189Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7688417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7688487Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7688717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7688842Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7688846Z 
2025-09-07T09:14:29.7688935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7689128Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7689187Z     return mod(**inputs)
2025-09-07T09:14:29.7689456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7689523Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7689790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7689859Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7690075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7690147Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7690375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7690444Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7690675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7690749Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7690978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7691129Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7691132Z 
2025-09-07T09:14:29.7691224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7691441Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7691502Z     return mod(**inputs)
2025-09-07T09:14:29.7691738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7691803Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7692065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7692133Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7692345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7692418Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7692650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7692726Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7692954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7693026Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7693256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7693328Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7693333Z 
2025-09-07T09:14:29.7693430Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7693616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7693675Z     return mod(**inputs)
2025-09-07T09:14:29.7693908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7693972Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7694203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7694268Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7694477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7694546Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7694773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7694848Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7695111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7695187Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7695446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7695549Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7695552Z 
2025-09-07T09:14:29.7695653Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7695841Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7695905Z     return mod(**inputs)
2025-09-07T09:14:29.7696139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7696205Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7696445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7696508Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7696724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7696795Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7697028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7697133Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7697364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7697442Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7697737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7697837Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7697841Z 
2025-09-07T09:14:29.7697932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7698115Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7698180Z     return mod(**inputs)
2025-09-07T09:14:29.7698409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7698479Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7698710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7698773Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7698989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7699057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7699284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7699356Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7699589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7699661Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7699885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7699983Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7699986Z 
2025-09-07T09:14:29.7700075Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7700264Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7700322Z     return mod(**inputs)
2025-09-07T09:14:29.7700551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7700654Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7700883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7700984Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7701196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7701264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7701499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7701570Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7701802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7701872Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7702101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7702171Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7702174Z 
2025-09-07T09:14:29.7702247Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7702339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7702523Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7702584Z     return mod(**inputs)
2025-09-07T09:14:29.7702845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7702910Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7703295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7703433Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7703660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7703730Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7703964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7704038Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7704271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7704379Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7704609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7704685Z     return self.weight * hidden_states
2025-09-07T09:14:29.7704689Z 
2025-09-07T09:14:29.7704783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7704978Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7705046Z     return mod(**inputs)
2025-09-07T09:14:29.7705278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7705353Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7705584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7705648Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7705866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7705936Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7706170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7706243Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7706471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7706631Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7706859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7706985Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7706988Z 
2025-09-07T09:14:29.7707080Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7707275Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7707331Z     return mod(**inputs)
2025-09-07T09:14:29.7707566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7707642Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7707873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7707944Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7708157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7708228Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7708464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7708535Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7708814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7708889Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7709125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7709229Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7709232Z 
2025-09-07T09:14:29.7709324Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7709517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7709576Z     return mod(**inputs)
2025-09-07T09:14:29.7709813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7709876Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7710108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7714330Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7714610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7714700Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7714963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7715045Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7715290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7715373Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7715610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7715739Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7715745Z 
2025-09-07T09:14:29.7715851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7716061Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7716123Z     return mod(**inputs)
2025-09-07T09:14:29.7716375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7716503Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7716740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7716815Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7717075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7717158Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7717398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7717475Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7717715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7717795Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7718040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7718192Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7718197Z 
2025-09-07T09:14:29.7718304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7718510Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7718573Z     return mod(**inputs)
2025-09-07T09:14:29.7718855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7718927Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7719172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7719237Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7719499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7719572Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7719801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7719875Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7720103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7720178Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7720404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7720474Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7720478Z 
2025-09-07T09:14:29.7720579Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7720772Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7720833Z     return mod(**inputs)
2025-09-07T09:14:29.7721064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7721127Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7721360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7721423Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7721642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7721710Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7721939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7722009Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7722235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7722344Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7722572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7722711Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7722715Z 
2025-09-07T09:14:29.7722808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7722995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7723058Z     return mod(**inputs)
2025-09-07T09:14:29.7723303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7723371Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7723601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7723663Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7723882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7723950Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7724184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7724253Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7724514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7724589Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7724815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7724911Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7724945Z 
2025-09-07T09:14:29.7725038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7725229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7725285Z     return mod(**inputs)
2025-09-07T09:14:29.7725517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7725583Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7725812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7725877Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7726092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7726160Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7726387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7726455Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7726683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7726754Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7726985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7727081Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7727084Z 
2025-09-07T09:14:29.7727175Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7727368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7727424Z     return mod(**inputs)
2025-09-07T09:14:29.7727659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7727725Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7727958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7728084Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7728296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7728415Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7728644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7728723Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7728950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7729022Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7729258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7729327Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7729332Z 
2025-09-07T09:14:29.7729413Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7729501Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7729685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7729749Z     return mod(**inputs)
2025-09-07T09:14:29.7729995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7730117Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7730354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7730418Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7730671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7730739Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7730974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7731059Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7731294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7731382Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7731615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7731689Z     return self.weight * hidden_states
2025-09-07T09:14:29.7731693Z 
2025-09-07T09:14:29.7731784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7731974Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7732034Z     return mod(**inputs)
2025-09-07T09:14:29.7732264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7732332Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7732560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7732628Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7732840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7732910Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7733142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7733222Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7733457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7733565Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7733841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7733915Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7733953Z 
2025-09-07T09:14:29.7734045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7734234Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7734292Z     return mod(**inputs)
2025-09-07T09:14:29.7734535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7734600Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7734830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7734900Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7735113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7735188Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7735413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7735495Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7735719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7735859Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7736097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7736171Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7736174Z 
2025-09-07T09:14:29.7736301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7736489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7736546Z     return mod(**inputs)
2025-09-07T09:14:29.7736782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7736848Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7737081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7737140Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7737358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7737428Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7737725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7737804Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7738033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7738135Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7738361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7738429Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7738433Z 
2025-09-07T09:14:29.7738526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7738710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7738767Z     return mod(**inputs)
2025-09-07T09:14:29.7738994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7739059Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7739290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7739538Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7739753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7739854Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7740083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7740165Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7740391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:14:29.7740515Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:14:29.7740519Z 
2025-09-07T09:14:29.7740592Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7740686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7740870Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7740926Z     return mod(**inputs)
2025-09-07T09:14:29.7741161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7741224Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7741454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7741547Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7741765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7741837Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7742096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7742173Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7742400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7742504Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7742729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7742798Z     return self.weight * hidden_states
2025-09-07T09:14:29.7742801Z 
2025-09-07T09:14:29.7742899Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7743083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7743147Z     return mod(**inputs)
2025-09-07T09:14:29.7743378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7743440Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7743675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7743741Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7743960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7744034Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7744262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7744339Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7744565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7744641Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7744869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7744940Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7744979Z 
2025-09-07T09:14:29.7745072Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7745254Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7745348Z     return mod(**inputs)
2025-09-07T09:14:29.7745577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7745644Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7745874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7745937Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7746150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7746217Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7746446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7746516Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7746746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7746822Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7747049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7747148Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7747152Z 
2025-09-07T09:14:29.7747245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7747428Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7747491Z     return mod(**inputs)
2025-09-07T09:14:29.7747748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7747814Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7748047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7748112Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7748327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7748391Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7748623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7748692Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7748918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7748995Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7749222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7749350Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7749354Z 
2025-09-07T09:14:29.7749441Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7749626Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7749683Z     return mod(**inputs)
2025-09-07T09:14:29.7749915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7749983Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7750211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7750275Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7750489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7750590Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7750822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7750940Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7751178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7751252Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7751479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7751630Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7751634Z 
2025-09-07T09:14:29.7751726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7751918Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7751977Z     return mod(**inputs)
2025-09-07T09:14:29.7752208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7752277Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7752507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7752575Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7752822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7752895Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7753122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7753193Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7753456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7753530Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7753761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7753835Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7753838Z 
2025-09-07T09:14:29.7753927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7754120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7754179Z     return mod(**inputs)
2025-09-07T09:14:29.7754416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7754480Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7754712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7754779Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7754994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7755066Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7755294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7755369Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7755599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7755671Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7755902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7756001Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7756005Z 
2025-09-07T09:14:29.7756100Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7756315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7756372Z     return mod(**inputs)
2025-09-07T09:14:29.7756611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7756706Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7756944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7757006Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7757225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7757292Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7757522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7757596Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7757824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7757901Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7758130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7758224Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7758227Z 
2025-09-07T09:14:29.7758359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7758546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7758607Z     return mod(**inputs)
2025-09-07T09:14:29.7758867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7758933Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7759171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7759234Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7759454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7759525Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7759755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7759828Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7760054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7760134Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7760363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7760462Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7760465Z 
2025-09-07T09:14:29.7760553Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7760736Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7760799Z     return mod(**inputs)
2025-09-07T09:14:29.7761031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7761099Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7761333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7761396Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7761615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7761682Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7761972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7762043Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7762276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7762380Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7762608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7762681Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7762685Z 
2025-09-07T09:14:29.7762759Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7762854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7763041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7763099Z     return mod(**inputs)
2025-09-07T09:14:29.7763337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7763401Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7763637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7763702Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7763912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7764022Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7764254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7764330Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7764603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7764705Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7764953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7765023Z     return self.weight * hidden_states
2025-09-07T09:14:29.7765028Z 
2025-09-07T09:14:29.7765128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7765321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7765384Z     return mod(**inputs)
2025-09-07T09:14:29.7765622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7765688Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7765924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7765988Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7766211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7766281Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7766513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7766589Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7766816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7766899Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7767126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7767199Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7767202Z 
2025-09-07T09:14:29.7767295Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7767483Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7767588Z     return mod(**inputs)
2025-09-07T09:14:29.7767819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7767923Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7768150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7768223Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7768437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7768505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7768732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7768802Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7769032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7769104Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7769333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7769400Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7769404Z 
2025-09-07T09:14:29.7769498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7769722Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7769782Z     return mod(**inputs)
2025-09-07T09:14:29.7770019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7770084Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7770348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7770412Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7770625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7770697Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7770924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7770998Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7771224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7771297Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7771524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7771645Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7771650Z 
2025-09-07T09:14:29.7771745Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7771929Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7771987Z     return mod(**inputs)
2025-09-07T09:14:29.7772214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7772277Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7772513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7772574Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7772785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7772851Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7773078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7773212Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7773441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7773550Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7773776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7773931Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7773935Z 
2025-09-07T09:14:29.7774021Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7774203Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7774262Z     return mod(**inputs)
2025-09-07T09:14:29.7774491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7774557Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7774785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7774846Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7775060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7775125Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7775388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7775459Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7775685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7775789Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7776020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7776096Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7776099Z 
2025-09-07T09:14:29.7776186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7776373Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7776430Z     return mod(**inputs)
2025-09-07T09:14:29.7776658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7776724Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7776953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7777018Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7777231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7777297Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7777573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7777644Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7777871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7777942Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7778169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7778270Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7778274Z 
2025-09-07T09:14:29.7778363Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7778552Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7778608Z     return mod(**inputs)
2025-09-07T09:14:29.7778881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7778942Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7779212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7779279Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7779494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7779562Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7779790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7779859Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7780094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7780166Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7780395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7780489Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7780493Z 
2025-09-07T09:14:29.7780582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7780767Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7780853Z     return mod(**inputs)
2025-09-07T09:14:29.7781088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7781149Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7781412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7781474Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7781686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7781756Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7781984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7782054Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7782280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7782351Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7782581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7782673Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7782678Z 
2025-09-07T09:14:29.7782771Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7782953Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7783011Z     return mod(**inputs)
2025-09-07T09:14:29.7783240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7783302Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7783542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7783604Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7783817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7783882Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7784115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7784193Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7784455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7784530Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7784792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7784862Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7784869Z 
2025-09-07T09:14:29.7784941Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7785031Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7785218Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7785273Z     return mod(**inputs)
2025-09-07T09:14:29.7785506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7785569Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7785799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7785862Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7786075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7786141Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7786402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7786486Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7786722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7786807Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7787068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7787136Z     return self.weight * hidden_states
2025-09-07T09:14:29.7787139Z 
2025-09-07T09:14:29.7787231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7787412Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7787466Z     return mod(**inputs)
2025-09-07T09:14:29.7787697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7787763Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7787994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7788052Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7788266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7788336Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7788562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7788642Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7788868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7788975Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7789205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7789273Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7789276Z 
2025-09-07T09:14:29.7789366Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7789550Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7789607Z     return mod(**inputs)
2025-09-07T09:14:29.7789874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7789938Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7790174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7790270Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7790485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7790553Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7790778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7790858Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7791083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7791188Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7791413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7791485Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7791493Z 
2025-09-07T09:14:29.7791582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7791764Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7791859Z     return mod(**inputs)
2025-09-07T09:14:29.7792091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7792155Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7792428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7792492Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7792708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7792772Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7793004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7793083Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7793308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7793413Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7793643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7793713Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7793716Z 
2025-09-07T09:14:29.7793786Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7793878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7794065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7794119Z     return mod(**inputs)
2025-09-07T09:14:29.7794350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7794414Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7794649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7794708Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7794919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7794988Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7795216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7795325Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7795551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7795648Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7795917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7795985Z     return self.weight * hidden_states
2025-09-07T09:14:29.7795988Z 
2025-09-07T09:14:29.7796083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7796265Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7796323Z     return mod(**inputs)
2025-09-07T09:14:29.7796553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7796616Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7796853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7796913Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7797126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7797193Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7797448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7797524Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7797748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7797824Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7798113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7798182Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7798187Z 
2025-09-07T09:14:29.7798276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7798459Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7798520Z     return mod(**inputs)
2025-09-07T09:14:29.7798747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7798813Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7799042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7799104Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7799319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7799382Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7799614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7799685Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7799910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7799987Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7800213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7800286Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7800289Z 
2025-09-07T09:14:29.7800375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7800554Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7800615Z     return mod(**inputs)
2025-09-07T09:14:29.7800843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7800948Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7801177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7801275Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7801488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7801553Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7801783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7801853Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7802081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7802157Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7802386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7802512Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7802517Z 
2025-09-07T09:14:29.7802606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7802793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7802848Z     return mod(**inputs)
2025-09-07T09:14:29.7803341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7803404Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7803635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7803744Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7803967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7804040Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7804272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7804347Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7804580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7804652Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7804882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7805030Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7805034Z 
2025-09-07T09:14:29.7805129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7805321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7805379Z     return mod(**inputs)
2025-09-07T09:14:29.7805619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7805687Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7805920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7805982Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7806200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7806271Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7806498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7806569Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7806795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7806914Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7807147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7807267Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7807269Z 
2025-09-07T09:14:29.7807363Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7807552Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7807607Z     return mod(**inputs)
2025-09-07T09:14:29.7807843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7807906Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7808142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7808204Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7808420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7808488Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7808713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7808787Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7809045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7809119Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7809345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7809479Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7809482Z 
2025-09-07T09:14:29.7809576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7809758Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7809815Z     return mod(**inputs)
2025-09-07T09:14:29.7810046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7810112Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7810342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7810401Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7810617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7810681Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7810909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7810977Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7811203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7811278Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7811501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7811593Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7811597Z 
2025-09-07T09:14:29.7811685Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7811867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7811926Z     return mod(**inputs)
2025-09-07T09:14:29.7812156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7812220Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7812486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7812553Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7812799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7812865Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7813095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7813165Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7813391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7813460Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7813688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7813788Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7813791Z 
2025-09-07T09:14:29.7813878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7814066Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7814120Z     return mod(**inputs)
2025-09-07T09:14:29.7814350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7814460Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7814691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7814755Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7815001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7815069Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7815297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7815364Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7815592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7815660Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7815893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7815961Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7815965Z 
2025-09-07T09:14:29.7816057Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7816242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7816300Z     return mod(**inputs)
2025-09-07T09:14:29.7816532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7816594Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7816824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7816891Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7817103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7817176Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7817403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7817471Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7817743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.7817869Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7817911Z 
2025-09-07T09:14:29.7817987Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7818076Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7818296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7818353Z     return mod(**inputs)
2025-09-07T09:14:29.7818584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7818652Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7818882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7818945Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7819160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7819227Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7819461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7819527Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7819762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7819859Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7820119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7820186Z     return self.weight * hidden_states
2025-09-07T09:14:29.7820189Z 
2025-09-07T09:14:29.7820278Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7820500Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7820556Z     return mod(**inputs)
2025-09-07T09:14:29.7820791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7820853Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7821085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7821159Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7821373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7821448Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7821677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7821752Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7821980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7822054Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7822287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7822356Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7822361Z 
2025-09-07T09:14:29.7822455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7822638Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7822694Z     return mod(**inputs)
2025-09-07T09:14:29.7822933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7822995Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7823229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7823290Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7823502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7823609Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7823837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7823945Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7824171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7824248Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7824479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7824546Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7824550Z 
2025-09-07T09:14:29.7824645Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7824828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7824888Z     return mod(**inputs)
2025-09-07T09:14:29.7825116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7825180Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7825412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7825472Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7825727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7825794Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7826029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7826133Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7826364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7826445Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7826672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7826794Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7826798Z 
2025-09-07T09:14:29.7826889Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7827077Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7827138Z     return mod(**inputs)
2025-09-07T09:14:29.7827368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7827436Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7827665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7827729Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7827950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7828021Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7828251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7828322Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7828551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7828624Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7828848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7829002Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7829039Z 
2025-09-07T09:14:29.7829127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7829316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7829404Z     return mod(**inputs)
2025-09-07T09:14:29.7829638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7829706Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7829935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7830001Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7830215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7830287Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7830515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7830584Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7830817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7830891Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7831120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7831220Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7831223Z 
2025-09-07T09:14:29.7831309Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7831498Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7831556Z     return mod(**inputs)
2025-09-07T09:14:29.7831821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7831885Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7832119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7832184Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7832396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7832466Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7832699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7832771Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7832999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7833075Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7833306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7833407Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7833410Z 
2025-09-07T09:14:29.7833505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7833691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7833748Z     return mod(**inputs)
2025-09-07T09:14:29.7833989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7834054Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7834286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7834346Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7834559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7834667Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7834895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7835018Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7835251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7835329Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7835559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7835651Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7835655Z 
2025-09-07T09:14:29.7835749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7835937Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7836001Z     return mod(**inputs)
2025-09-07T09:14:29.7836236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7836298Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7836536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7836600Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7836869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7836944Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7837175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7837249Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7837506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7837589Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7837818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7837916Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7837919Z 
2025-09-07T09:14:29.7838009Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7838192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7838258Z     return mod(**inputs)
2025-09-07T09:14:29.7838488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7838555Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7838785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7838849Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7839066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7839134Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7839366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7839438Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7839669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7839741Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7839969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7840040Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7840045Z 
2025-09-07T09:14:29.7840117Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7840242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7840425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7840482Z     return mod(**inputs)
2025-09-07T09:14:29.7840755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7840819Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7841062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7841123Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7841338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7841410Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7841639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7841729Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7841954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7842049Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7842277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7842347Z     return self.weight * hidden_states
2025-09-07T09:14:29.7842350Z 
2025-09-07T09:14:29.7842475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7842659Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7842722Z     return mod(**inputs)
2025-09-07T09:14:29.7842985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7843052Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7843291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7843356Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7843573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7843639Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7843867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7843950Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7844177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7844292Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7844518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7844595Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7844598Z 
2025-09-07T09:14:29.7844686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7844870Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7844932Z     return mod(**inputs)
2025-09-07T09:14:29.7845162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7845230Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7845458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7845519Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7845739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7845808Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7846080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7846162Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7846426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7846540Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7846775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7846852Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7846855Z 
2025-09-07T09:14:29.7846944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7847134Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7847192Z     return mod(**inputs)
2025-09-07T09:14:29.7847430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7847499Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7847734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7847804Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7848018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7848121Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7848358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7848440Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7848710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7848817Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7849051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7849126Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7849129Z 
2025-09-07T09:14:29.7849202Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7849302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7849494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7849556Z     return mod(**inputs)
2025-09-07T09:14:29.7849793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7849860Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7850099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7850164Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7850390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7850462Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7850692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7850770Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7850996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7851102Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7851329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7851405Z     return self.weight * hidden_states
2025-09-07T09:14:29.7851409Z 
2025-09-07T09:14:29.7851499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7851725Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7851787Z     return mod(**inputs)
2025-09-07T09:14:29.7852055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7852125Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7852360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7852424Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7852645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7852714Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7852946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7853017Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7853244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7853323Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7853550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7853625Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7853629Z 
2025-09-07T09:14:29.7853750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7853942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7854002Z     return mod(**inputs)
2025-09-07T09:14:29.7854265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7854337Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7854569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7854636Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7854848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7854919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7855152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7855225Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7855458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7855532Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7855761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7855836Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7855840Z 
2025-09-07T09:14:29.7855931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7856120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7856180Z     return mod(**inputs)
2025-09-07T09:14:29.7856415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7856478Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7856711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7856782Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7856993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7857068Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7857295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7857416Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7857743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7857852Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7858082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7858205Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7858209Z 
2025-09-07T09:14:29.7858306Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7858491Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7858551Z     return mod(**inputs)
2025-09-07T09:14:29.7858785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7858849Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7859084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7859150Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7859361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7859435Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7859698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7859777Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7860037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7860111Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7860343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7860487Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7860493Z 
2025-09-07T09:14:29.7860585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7860767Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7860827Z     return mod(**inputs)
2025-09-07T09:14:29.7861060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7861122Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7861359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7861431Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7861648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7861717Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7861940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7862019Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7862244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7862319Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7862544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7862613Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7862622Z 
2025-09-07T09:14:29.7862712Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7862896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7862993Z     return mod(**inputs)
2025-09-07T09:14:29.7863234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7863338Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7863571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7863632Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7863852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7863919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7864149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7864225Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7864457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7864535Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7864762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7864866Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7864869Z 
2025-09-07T09:14:29.7864958Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7865176Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7865236Z     return mod(**inputs)
2025-09-07T09:14:29.7865466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7865535Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7865805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7865878Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7866090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7866162Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7866399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7866471Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7866704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7866775Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7867000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7867104Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7867107Z 
2025-09-07T09:14:29.7867201Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7867390Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7867449Z     return mod(**inputs)
2025-09-07T09:14:29.7867687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7867752Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7867983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7868051Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7868266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7868336Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7868566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7868671Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7868902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7869008Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7869241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7869333Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7869338Z 
2025-09-07T09:14:29.7869433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7869618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7869678Z     return mod(**inputs)
2025-09-07T09:14:29.7869914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7869976Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7870214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7870275Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7870489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7870560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7870823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7870904Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7871139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7871213Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7871475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7871547Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7871550Z 
2025-09-07T09:14:29.7871631Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7871721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7871923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7871981Z     return mod(**inputs)
2025-09-07T09:14:29.7872213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7872283Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7872513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7872580Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7872794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7872864Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7873098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7873173Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7873405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7873500Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7873728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7873800Z     return self.weight * hidden_states
2025-09-07T09:14:29.7873803Z 
2025-09-07T09:14:29.7873894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7874084Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7874142Z     return mod(**inputs)
2025-09-07T09:14:29.7874420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7874486Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7874755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7874823Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7875038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7875115Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7875342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7875417Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7875650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7875725Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7875961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7876032Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7876036Z 
2025-09-07T09:14:29.7876135Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7876321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7876410Z     return mod(**inputs)
2025-09-07T09:14:29.7876649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7876711Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7876993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7877059Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7877275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7877353Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7877581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7877657Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7877885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7877959Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7878190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7878259Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7878263Z 
2025-09-07T09:14:29.7878360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7878544Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7878609Z     return mod(**inputs)
2025-09-07T09:14:29.7878842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7878907Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7879143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7879207Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7879427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7879495Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7879723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7879798Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7880062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7880141Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7880403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7880526Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7880533Z 
2025-09-07T09:14:29.7880624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7880810Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7880873Z     return mod(**inputs)
2025-09-07T09:14:29.7881102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7881174Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7881405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7881471Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7881692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7881761Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7881993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7882097Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7882329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7882405Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7882663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7882816Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7882821Z 
2025-09-07T09:14:29.7882909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7883100Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7883161Z     return mod(**inputs)
2025-09-07T09:14:29.7883391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7883456Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7883687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7883755Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7883965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7884035Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7884266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7884337Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7884565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7884639Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7884866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7884951Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7884954Z 
2025-09-07T09:14:29.7885042Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7885231Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7885288Z     return mod(**inputs)
2025-09-07T09:14:29.7885522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7885620Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7885854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7885951Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7886164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7886234Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7886462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7886535Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7886764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7886837Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7887068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7887164Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7887167Z 
2025-09-07T09:14:29.7887264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7887456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7887512Z     return mod(**inputs)
2025-09-07T09:14:29.7887813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7887880Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7888119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7888213Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7888429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7888502Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7888732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7888813Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7889042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7889118Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7889353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7889445Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7889448Z 
2025-09-07T09:14:29.7889542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7889732Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7889792Z     return mod(**inputs)
2025-09-07T09:14:29.7890029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7890094Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7890334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7890395Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7890616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7890683Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7890910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7890991Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7891219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7891333Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7891561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7891688Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7891691Z 
2025-09-07T09:14:29.7891789Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7891974Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7892039Z     return mod(**inputs)
2025-09-07T09:14:29.7892270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7892341Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7892573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7892638Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7892856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7892928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7893161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7893230Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7893491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7893573Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7893803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7893908Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7893911Z 
2025-09-07T09:14:29.7894002Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7894187Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7894249Z     return mod(**inputs)
2025-09-07T09:14:29.7894480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7894547Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7894778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7894842Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7895053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7895119Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7895356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7895429Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7895663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:14:29.7895789Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.7895792Z 
2025-09-07T09:14:29.7895866Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7895963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7896150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7896217Z     return mod(**inputs)
2025-09-07T09:14:29.7896448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7896516Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7896754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7896854Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7897078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7897183Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7897421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7897540Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7897772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7897865Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7898089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7898162Z     return self.weight * hidden_states
2025-09-07T09:14:29.7898165Z 
2025-09-07T09:14:29.7898258Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7898442Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7898504Z     return mod(**inputs)
2025-09-07T09:14:29.7898739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7898806Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7899081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7899144Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7899361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7899430Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7899692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7899772Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7900000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7900111Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7900336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7900410Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7900415Z 
2025-09-07T09:14:29.7900503Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7900692Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7900747Z     return mod(**inputs)
2025-09-07T09:14:29.7900977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7901047Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7901280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7901347Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7901562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7901629Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7901859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7901935Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7902162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7902263Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7902495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7902606Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7902609Z 
2025-09-07T09:14:29.7902698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7902921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7902980Z     return mod(**inputs)
2025-09-07T09:14:29.7903352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7903418Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7903651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7903721Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7903938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7904015Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7904248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7904329Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7904563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7904668Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7904960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7905035Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7905039Z 
2025-09-07T09:14:29.7905113Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7905202Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7905433Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7905501Z     return mod(**inputs)
2025-09-07T09:14:29.7905734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7905803Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7906036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7906099Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7906325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7906400Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7906633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7906706Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7906935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7907036Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7907260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7907334Z     return self.weight * hidden_states
2025-09-07T09:14:29.7907337Z 
2025-09-07T09:14:29.7907425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7907615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7907673Z     return mod(**inputs)
2025-09-07T09:14:29.7907902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7907968Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7908198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7908315Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7908528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7908595Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7908876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7908951Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7909185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7909259Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7909488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7909560Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7909563Z 
2025-09-07T09:14:29.7909656Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7909844Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7909901Z     return mod(**inputs)
2025-09-07T09:14:29.7910139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7910204Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7910437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7910540Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7910757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7910834Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7911097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7911173Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7911407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7911481Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7911716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7911786Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7911790Z 
2025-09-07T09:14:29.7911887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7912074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7912133Z     return mod(**inputs)
2025-09-07T09:14:29.7912375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7912438Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7912680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7912745Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7912959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7913032Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7913259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7913337Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7913565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7913645Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7913874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7913995Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7914033Z 
2025-09-07T09:14:29.7914129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7914312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7914407Z     return mod(**inputs)
2025-09-07T09:14:29.7914638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7914702Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7914938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7915001Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7915217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7915286Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7915515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7915589Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7915816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7915893Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7916117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7916322Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7916326Z 
2025-09-07T09:14:29.7916419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7916604Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7916708Z     return mod(**inputs)
2025-09-07T09:14:29.7916941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7917013Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7917243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7917313Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7917530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7917599Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7917833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7917904Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7918133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7918212Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7918439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7918517Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7918522Z 
2025-09-07T09:14:29.7918613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7918800Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7918858Z     return mod(**inputs)
2025-09-07T09:14:29.7919092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7919159Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7919392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7919461Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7919674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7919781Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7920014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7920135Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7920372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7920443Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7920673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7920775Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7920778Z 
2025-09-07T09:14:29.7920869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7921056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7921117Z     return mod(**inputs)
2025-09-07T09:14:29.7921359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7921425Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7921658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7921726Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7921970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7922044Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7922272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7922374Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7922613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7922687Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7922923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7923019Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7923024Z 
2025-09-07T09:14:29.7923120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7923308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7923366Z     return mod(**inputs)
2025-09-07T09:14:29.7923605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7923669Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7923909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7923973Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7924187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7924261Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7924490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7924562Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7924790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7924863Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7925097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7925190Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7925193Z 
2025-09-07T09:14:29.7925325Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7925509Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7925571Z     return mod(**inputs)
2025-09-07T09:14:29.7925842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7925908Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7926150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7926212Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7926431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7926503Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7926731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7926811Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7927036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7927110Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7927339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7927412Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7927415Z 
2025-09-07T09:14:29.7927522Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7927613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7927802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7927862Z     return mod(**inputs)
2025-09-07T09:14:29.7928134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7931412Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7931711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7931793Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7932035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7932119Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7932366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7932439Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7932698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7932808Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7933041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7933114Z     return self.weight * hidden_states
2025-09-07T09:14:29.7933119Z 
2025-09-07T09:14:29.7933222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7933434Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7933499Z     return mod(**inputs)
2025-09-07T09:14:29.7933750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7933822Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7934068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7934142Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7934369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7934509Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7934743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7934857Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7935085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7935159Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7935390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7935461Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7935464Z 
2025-09-07T09:14:29.7935565Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7935765Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7935826Z     return mod(**inputs)
2025-09-07T09:14:29.7936062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7936129Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7936366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7936427Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7936673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7936746Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7936983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7937057Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7937316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7937392Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7937689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7937759Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7937762Z 
2025-09-07T09:14:29.7937859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7938046Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7938109Z     return mod(**inputs)
2025-09-07T09:14:29.7938343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7938406Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7938645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7938706Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7938927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7938992Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7939221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7939299Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7939528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7939602Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7939828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7939951Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7939956Z 
2025-09-07T09:14:29.7940045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7940272Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7940333Z     return mod(**inputs)
2025-09-07T09:14:29.7940566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7940667Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7940898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7940960Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7941178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7941242Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7941472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7941541Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7941773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7941845Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7942072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7942229Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7942234Z 
2025-09-07T09:14:29.7942353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7942546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7942602Z     return mod(**inputs)
2025-09-07T09:14:29.7942867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7942937Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7943168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7943234Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7943445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7943513Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7943750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7943822Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7944053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7944124Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7944358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7944430Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7944434Z 
2025-09-07T09:14:29.7944522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7944707Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7944765Z     return mod(**inputs)
2025-09-07T09:14:29.7944998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7945059Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7945293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7945358Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7945568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7945638Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7945862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7945989Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7946215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7946323Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7946553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7946657Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7946661Z 
2025-09-07T09:14:29.7946759Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7946942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7947000Z     return mod(**inputs)
2025-09-07T09:14:29.7947234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7947299Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7947532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7947595Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7947805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7947878Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7948138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7948217Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7948475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7948553Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7948781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7948874Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7948879Z 
2025-09-07T09:14:29.7948975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7949158Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7949222Z     return mod(**inputs)
2025-09-07T09:14:29.7949451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7949512Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7949743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7949803Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7950020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7950089Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7950318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7950389Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7950612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7950689Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7950911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7951005Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7951008Z 
2025-09-07T09:14:29.7951096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7951278Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7951372Z     return mod(**inputs)
2025-09-07T09:14:29.7951604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7951701Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7951930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7951991Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7952205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7952270Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7952497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7952566Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7952792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7952863Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7953087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7953159Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7953162Z 
2025-09-07T09:14:29.7953236Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7953327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7953537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7953595Z     return mod(**inputs)
2025-09-07T09:14:29.7953828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7953921Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7954157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7954221Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7954432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7954502Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7954727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7954815Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7955043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.7955132Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7955362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7955430Z     return self.weight * hidden_states
2025-09-07T09:14:29.7955435Z 
2025-09-07T09:14:29.7955527Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7955712Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7955772Z     return mod(**inputs)
2025-09-07T09:14:29.7956000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7956077Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7956308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7956369Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7956583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7956650Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7956877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7956992Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7957217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7957594Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7957820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.7957896Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.7957900Z 
2025-09-07T09:14:29.7957988Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7958170Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7958228Z     return mod(**inputs)
2025-09-07T09:14:29.7958458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7958523Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7958754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7958815Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7959031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7959095Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7959359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7959438Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7959666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7959803Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7960034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.7960118Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.7960121Z 
2025-09-07T09:14:29.7960212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7960405Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7960463Z     return mod(**inputs)
2025-09-07T09:14:29.7960696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7960766Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7960996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7961064Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7961280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7961348Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7961585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7961663Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7961901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.7962003Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.7962240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.7962311Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.7962314Z 
2025-09-07T09:14:29.7962407Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7962605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7962706Z     return mod(**inputs)
2025-09-07T09:14:29.7962947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7963015Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7963282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7963356Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7963570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7963646Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7963883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.7963963Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.7964198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:14:29.7964322Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:14:29.7964325Z 
2025-09-07T09:14:29.7964401Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7964498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7964683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7964748Z     return mod(**inputs)
2025-09-07T09:14:29.7965011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7965075Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7965313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7965426Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7965648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7965720Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7965955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7966032Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7966260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.7966365Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7966593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7966670Z     return self.weight * hidden_states
2025-09-07T09:14:29.7966673Z 
2025-09-07T09:14:29.7966764Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7966948Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7967011Z     return mod(**inputs)
2025-09-07T09:14:29.7967241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7967314Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7967544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7967607Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7967829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7967895Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7968128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7968198Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7968424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7968531Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7968759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7968866Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7968869Z 
2025-09-07T09:14:29.7968956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7969145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7969203Z     return mod(**inputs)
2025-09-07T09:14:29.7969433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7969499Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7969730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7969797Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7970008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7970074Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7970303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7970372Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7970635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7970707Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7970935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7971002Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7971037Z 
2025-09-07T09:14:29.7971127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7971314Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7971369Z     return mod(**inputs)
2025-09-07T09:14:29.7971599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7971663Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7971894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7971961Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7972169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7972239Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7972467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7972539Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7972765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7972833Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7973065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7973185Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7973189Z 
2025-09-07T09:14:29.7973281Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7973463Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7973520Z     return mod(**inputs)
2025-09-07T09:14:29.7973758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7973822Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7974095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7974157Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7974370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7974472Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7974700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7974775Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7975007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7975079Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7975308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7975464Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7975467Z 
2025-09-07T09:14:29.7975560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7975756Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7975815Z     return mod(**inputs)
2025-09-07T09:14:29.7976061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7976157Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7976395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7976455Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7976705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7976783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7977015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7977085Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7977312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7977388Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7977673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7977744Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7977748Z 
2025-09-07T09:14:29.7977834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7978019Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7978079Z     return mod(**inputs)
2025-09-07T09:14:29.7978313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7978378Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7978605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7978669Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7978885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7978952Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7979182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7979252Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7979483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7979552Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7979818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7979921Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7979989Z 
2025-09-07T09:14:29.7980078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7980266Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7980320Z     return mod(**inputs)
2025-09-07T09:14:29.7980549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7980617Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7980851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7980915Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7981125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7981194Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7981424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7981494Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7981724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7981824Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7982069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.7982165Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.7982168Z 
2025-09-07T09:14:29.7982289Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7982494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7982556Z     return mod(**inputs)
2025-09-07T09:14:29.7982796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7982861Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7983092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7983158Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7983372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7983441Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7983668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7983738Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7983968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7984041Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7984270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.7984363Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.7984367Z 
2025-09-07T09:14:29.7984461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7984646Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7984701Z     return mod(**inputs)
2025-09-07T09:14:29.7984933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7985007Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7985239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7985332Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7985554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7985652Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7985884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.7985954Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.7986181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.7986255Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.7986480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.7986552Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.7986556Z 
2025-09-07T09:14:29.7986627Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.7986720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7986904Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7986965Z     return mod(**inputs)
2025-09-07T09:14:29.7987197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7987259Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7987537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7987599Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7987813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7987919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7988151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7988228Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7988454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.7988554Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.7988785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.7988855Z     return self.weight * hidden_states
2025-09-07T09:14:29.7988859Z 
2025-09-07T09:14:29.7988952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7989136Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7989194Z     return mod(**inputs)
2025-09-07T09:14:29.7989425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7989491Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7989722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7989785Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7990000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7990067Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7990295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7990369Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7990593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7990667Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7990893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.7990997Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.7991002Z 
2025-09-07T09:14:29.7991093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7991310Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7991368Z     return mod(**inputs)
2025-09-07T09:14:29.7991598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7991663Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7991892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7991952Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7992169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7992235Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7992466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7992538Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7992764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7992837Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7993099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.7993170Z     key_states = self.k(current_states)
2025-09-07T09:14:29.7993174Z 
2025-09-07T09:14:29.7993260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7993478Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7993534Z     return mod(**inputs)
2025-09-07T09:14:29.7993766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7993832Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7994063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7994128Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7994342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7994408Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7994643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7994711Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7994941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7995014Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7995240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.7995363Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.7995366Z 
2025-09-07T09:14:29.7995456Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7995642Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7995698Z     return mod(**inputs)
2025-09-07T09:14:29.7995933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7995994Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7996222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7996285Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7996537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7996608Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7996870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7996938Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7997169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7997240Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7997470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.7997618Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.7997622Z 
2025-09-07T09:14:29.7997711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.7997897Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.7997953Z     return mod(**inputs)
2025-09-07T09:14:29.7998185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.7998249Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.7998478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.7998572Z     layer_outputs = layer_module(
2025-09-07T09:14:29.7998788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.7998855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.7999115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.7999190Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.7999418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.7999489Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.7999722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.7999790Z     value_states = self.v(current_states)
2025-09-07T09:14:29.7999793Z 
2025-09-07T09:14:29.7999887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8000068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8000126Z     return mod(**inputs)
2025-09-07T09:14:29.8000356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8000417Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8000650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8000710Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8000922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8000990Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8001219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8001293Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8001517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8001593Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8001817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8001952Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8001958Z 
2025-09-07T09:14:29.8002051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8002233Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8002326Z     return mod(**inputs)
2025-09-07T09:14:29.8002556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8002623Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8002853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8002914Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8003313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8003382Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8003625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8003699Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8003926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8004005Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8004229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8004392Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8004396Z 
2025-09-07T09:14:29.8004490Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8004682Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8004784Z     return mod(**inputs)
2025-09-07T09:14:29.8005027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8005100Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8005332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8005401Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8005621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8005691Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8005931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8006001Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8006235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8006305Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8006534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.8006637Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.8006642Z 
2025-09-07T09:14:29.8006738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8006933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8006990Z     return mod(**inputs)
2025-09-07T09:14:29.8007231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8007298Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8007528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8007596Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8007810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8007929Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8008157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8008288Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8008522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8008595Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8008821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.8008888Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.8008892Z 
2025-09-07T09:14:29.8008966Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8009062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8009253Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8009316Z     return mod(**inputs)
2025-09-07T09:14:29.8009543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8009610Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8009838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8009898Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8010155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8010221Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8010485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8010571Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8010798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.8010889Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8011118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8011187Z     return self.weight * hidden_states
2025-09-07T09:14:29.8011191Z 
2025-09-07T09:14:29.8011283Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8011469Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8011532Z     return mod(**inputs)
2025-09-07T09:14:29.8011761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8011827Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8012059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8012127Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8012342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8012411Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8012644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8012725Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8012959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8013068Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8013294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.8013373Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.8013410Z 
2025-09-07T09:14:29.8013500Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8013691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8013783Z     return mod(**inputs)
2025-09-07T09:14:29.8014015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8014084Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8014316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8014379Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8014589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8014658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8014885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8014964Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8015197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8015304Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8015532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.8015641Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.8015645Z 
2025-09-07T09:14:29.8015736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8015928Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8015982Z     return mod(**inputs)
2025-09-07T09:14:29.8016251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8016315Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8016551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8016616Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8016827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8016899Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8017127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8017207Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8017434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8017563Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8017797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.8017867Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.8017870Z 
2025-09-07T09:14:29.8017948Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8018035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8018219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8018280Z     return mod(**inputs)
2025-09-07T09:14:29.8018511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8018575Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8018803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8018868Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8019081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8019184Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8019416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8019522Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8019754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.8019853Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8020079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8020150Z     return self.weight * hidden_states
2025-09-07T09:14:29.8020153Z 
2025-09-07T09:14:29.8020241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8020425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8020481Z     return mod(**inputs)
2025-09-07T09:14:29.8020709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8020774Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8021000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8021062Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8021304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8021375Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8021602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8021703Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8021932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8022006Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8022233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.8022302Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.8022306Z 
2025-09-07T09:14:29.8022393Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8022579Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8022634Z     return mod(**inputs)
2025-09-07T09:14:29.8022871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8022933Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8023167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8023232Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8023447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8023520Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8023748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8023824Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8024050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8024123Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8024353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.8024423Z     key_states = self.k(current_states)
2025-09-07T09:14:29.8024426Z 
2025-09-07T09:14:29.8024557Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8024743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8024801Z     return mod(**inputs)
2025-09-07T09:14:29.8025072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8025136Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8025375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8025445Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8025666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8025737Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8025966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8026043Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8026280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8026354Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8026584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.8026705Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.8026742Z 
2025-09-07T09:14:29.8026836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8027021Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8027080Z     return mod(**inputs)
2025-09-07T09:14:29.8027346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8027410Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8027644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8027706Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8027925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8027990Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8028222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8028292Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8028517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8028597Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8028822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.8028973Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.8028977Z 
2025-09-07T09:14:29.8029064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8029251Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8029312Z     return mod(**inputs)
2025-09-07T09:14:29.8029540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8029604Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8029832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8029894Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8030113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8030231Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8030463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8030533Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8030801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8030871Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8031100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.8031172Z     value_states = self.v(current_states)
2025-09-07T09:14:29.8031175Z 
2025-09-07T09:14:29.8031262Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8031451Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8031508Z     return mod(**inputs)
2025-09-07T09:14:29.8031738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8031804Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8032037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8032102Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8032312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8032413Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8032644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8032715Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8032982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8033052Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8033284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8033382Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8033387Z 
2025-09-07T09:14:29.8033475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8033659Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8033715Z     return mod(**inputs)
2025-09-07T09:14:29.8033945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8034006Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8034234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8034299Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8034514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8034580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8034807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8034880Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8035112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8035182Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8035411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8035500Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8035503Z 
2025-09-07T09:14:29.8035595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8035813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8035868Z     return mod(**inputs)
2025-09-07T09:14:29.8036100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8036193Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8036428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8036491Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8036702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8036770Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8036997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8037070Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8037294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8037363Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8037590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.8037681Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.8037685Z 
2025-09-07T09:14:29.8037775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8037990Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8038054Z     return mod(**inputs)
2025-09-07T09:14:29.8038285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8038380Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8038616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8038679Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8038899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8038967Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8039201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8039271Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8039496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8039572Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8039800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.8039875Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.8039879Z 
2025-09-07T09:14:29.8039967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8040150Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8040215Z     return mod(**inputs)
2025-09-07T09:14:29.8040444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8040510Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8040743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8040807Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8041024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8041091Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8041325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8041430Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8041663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:14:29.8041825Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.8041828Z 
2025-09-07T09:14:29.8041900Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8041996Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8042182Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8042245Z     return mod(**inputs)
2025-09-07T09:14:29.8042475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8042541Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8042774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8042837Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8043052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8043120Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8043351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8043461Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8043691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.8043794Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8044054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8044129Z     return self.weight * hidden_states
2025-09-07T09:14:29.8044133Z 
2025-09-07T09:14:29.8044224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8044407Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8044470Z     return mod(**inputs)
2025-09-07T09:14:29.8044702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8044771Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8045003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8045067Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8045286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8045364Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8045597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8045669Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8045898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8045977Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8046216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.8046298Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.8046302Z 
2025-09-07T09:14:29.8046391Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8046577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8046636Z     return mod(**inputs)
2025-09-07T09:14:29.8046867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8046972Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8047206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8047303Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8047521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8047589Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8047822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8047891Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8048123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8048197Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8048423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.8048497Z     key_states = self.k(current_states)
2025-09-07T09:14:29.8048501Z 
2025-09-07T09:14:29.8048591Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8048782Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8048841Z     return mod(**inputs)
2025-09-07T09:14:29.8049106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8049169Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8049401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8049467Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8049723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8049800Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8050029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8050100Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8050332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8050405Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8050635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.8050751Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.8050755Z 
2025-09-07T09:14:29.8050845Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8051028Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8051097Z     return mod(**inputs)
2025-09-07T09:14:29.8051337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8051400Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8051635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8051697Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8051910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8051981Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8052210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8052283Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8052509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8052618Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8052850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.8053027Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.8053030Z 
2025-09-07T09:14:29.8053125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8053311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8053372Z     return mod(**inputs)
2025-09-07T09:14:29.8053601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8053662Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8053907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8053971Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8054185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8054252Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8054482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8054556Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8054816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8054895Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8055122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.8055193Z     value_states = self.v(current_states)
2025-09-07T09:14:29.8055232Z 
2025-09-07T09:14:29.8055322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8055505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8055565Z     return mod(**inputs)
2025-09-07T09:14:29.8055795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8055866Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8056095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8056156Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8056373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8056438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8056673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8056746Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8056974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8057051Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8057277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8057380Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8057384Z 
2025-09-07T09:14:29.8057472Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8057699Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8057756Z     return mod(**inputs)
2025-09-07T09:14:29.8057988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8058056Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8058326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8058393Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8058605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8058706Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8058941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8059015Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8059249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8059321Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8059548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8059648Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8059652Z 
2025-09-07T09:14:29.8059740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8059926Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8059988Z     return mod(**inputs)
2025-09-07T09:14:29.8060228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8060322Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8060558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8060625Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8060870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8060940Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8061174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8061244Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8061474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8061552Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8061785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.8061876Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.8061879Z 
2025-09-07T09:14:29.8061974Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8062159Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8062214Z     return mod(**inputs)
2025-09-07T09:14:29.8062447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8062511Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8062743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8062808Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8063024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8063097Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8063322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8063395Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8063622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8063693Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8063967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.8064035Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.8064071Z 
2025-09-07T09:14:29.8064146Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8064237Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8064424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8064483Z     return mod(**inputs)
2025-09-07T09:14:29.8064718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8064783Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8065013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8065077Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8065290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8065357Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8065589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8065673Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8065935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.8066021Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8066247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8066318Z     return self.weight * hidden_states
2025-09-07T09:14:29.8066321Z 
2025-09-07T09:14:29.8066442Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8066631Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8066690Z     return mod(**inputs)
2025-09-07T09:14:29.8066924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8066988Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8067217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8067282Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8067493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8067564Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8067792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8067869Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8068102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8068209Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8068442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.8068512Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.8068516Z 
2025-09-07T09:14:29.8068604Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8068792Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8068848Z     return mod(**inputs)
2025-09-07T09:14:29.8069080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8069143Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8069374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8069496Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8069710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8069814Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8070044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8070126Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8070357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8070459Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8070692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.8070766Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.8070771Z 
2025-09-07T09:14:29.8070865Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8071047Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8071109Z     return mod(**inputs)
2025-09-07T09:14:29.8071337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8071399Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8071682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8071744Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8071964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8072064Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8072293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8072375Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8072602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8072709Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8072933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.8073004Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.8073012Z 
2025-09-07T09:14:29.8073083Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8073170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8073356Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8073414Z     return mod(**inputs)
2025-09-07T09:14:29.8073651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8073715Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8073944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8074013Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8074226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8074295Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8074521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8074591Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8074823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:14:29.8074919Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8075187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8075255Z     return self.weight * hidden_states
2025-09-07T09:14:29.8075292Z 
2025-09-07T09:14:29.8075381Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8075569Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8075625Z     return mod(**inputs)
2025-09-07T09:14:29.8075861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8075924Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8076158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8076220Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8076432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8076505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8076731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8076807Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8077037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8077141Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8077375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.8077443Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.8077446Z 
2025-09-07T09:14:29.8077573Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8077757Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8077815Z     return mod(**inputs)
2025-09-07T09:14:29.8078050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8078115Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8078350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8078413Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8078626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8078694Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8078921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8079000Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8079226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8079302Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8079527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.8079596Z     key_states = self.k(current_states)
2025-09-07T09:14:29.8079600Z 
2025-09-07T09:14:29.8079693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8079877Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8079940Z     return mod(**inputs)
2025-09-07T09:14:29.8080171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8080235Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8080474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8080570Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8080788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8080887Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8081125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8081198Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8081428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8081507Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8081735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.8081858Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.8081862Z 
2025-09-07T09:14:29.8081949Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8082134Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8082194Z     return mod(**inputs)
2025-09-07T09:14:29.8082425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8082494Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8082756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8082824Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8083036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8083103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8083366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8083438Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8083671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8083743Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8083969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.8084118Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.8084122Z 
2025-09-07T09:14:29.8084211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8084399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8084453Z     return mod(**inputs)
2025-09-07T09:14:29.8084684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8084751Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8084978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8085044Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8085258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8085330Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8085560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8085628Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8085855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8085926Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8086158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.8086261Z     value_states = self.v(current_states)
2025-09-07T09:14:29.8086264Z 
2025-09-07T09:14:29.8086351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8086571Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8086629Z     return mod(**inputs)
2025-09-07T09:14:29.8086870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8086936Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8087166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8087233Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8087447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8087518Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8087748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8087820Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8088051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8088120Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8088382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8088477Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8088480Z 
2025-09-07T09:14:29.8088572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8088787Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8088843Z     return mod(**inputs)
2025-09-07T09:14:29.8089081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8089145Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8089378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8089440Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8089653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8089723Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8089950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8090024Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8090252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8090337Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8090586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8090679Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8090685Z 
2025-09-07T09:14:29.8090777Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8090957Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8091025Z     return mod(**inputs)
2025-09-07T09:14:29.8091253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8091314Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8091549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8091612Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8091877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8091945Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8092182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8092309Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8092538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8092611Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8092838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.8092940Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.8092943Z 
2025-09-07T09:14:29.8093036Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8093224Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8093285Z     return mod(**inputs)
2025-09-07T09:14:29.8093517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8093587Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8093818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8093911Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8094137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8094203Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8094465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:14:29.8094537Z     self_attention_outputs = self.layer[0](
2025-09-07T09:14:29.8094769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:14:29.8094837Z     attention_output = self.SelfAttention(
2025-09-07T09:14:29.8095063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.8095135Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.8095138Z 
2025-09-07T09:14:29.8095208Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8095304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8095489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8095544Z     return mod(**inputs)
2025-09-07T09:14:29.8095780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8095843Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8096076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8096136Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8096349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8096424Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8096651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8096725Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8096952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:14:29.8097052Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8097277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8097377Z     return self.weight * hidden_states
2025-09-07T09:14:29.8097380Z 
2025-09-07T09:14:29.8097470Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8097727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8097820Z     return mod(**inputs)
2025-09-07T09:14:29.8098051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8098117Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8098351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8098410Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8098624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8098689Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8098919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8098989Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8099212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8099296Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8099561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:14:29.8099631Z     query_states = self.q(hidden_states)
2025-09-07T09:14:29.8099635Z 
2025-09-07T09:14:29.8099721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8099909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8100005Z     return mod(**inputs)
2025-09-07T09:14:29.8100242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8100310Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8100541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8100603Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8100823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8100890Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8101118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8101188Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8101419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8101490Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8101714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:14:29.8101786Z     key_states = self.k(current_states)
2025-09-07T09:14:29.8101789Z 
2025-09-07T09:14:29.8101881Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8102067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8102122Z     return mod(**inputs)
2025-09-07T09:14:29.8102355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8102422Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8102653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8102715Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8102926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8103026Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8103407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8103531Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8103769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8103840Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8104072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:14:29.8104195Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:14:29.8104199Z 
2025-09-07T09:14:29.8104286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8104474Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8104533Z     return mod(**inputs)
2025-09-07T09:14:29.8104766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8104830Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8105058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8105124Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8105388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8105458Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8105684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8105807Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8106039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8106112Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8106338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:14:29.8106488Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:14:29.8106492Z 
2025-09-07T09:14:29.8106583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8106767Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8106822Z     return mod(**inputs)
2025-09-07T09:14:29.8107059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8107123Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8107354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8107416Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8107628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8107700Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8107925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8107997Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8108222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8108298Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8108523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:14:29.8108591Z     value_states = self.v(current_states)
2025-09-07T09:14:29.8108594Z 
2025-09-07T09:14:29.8108733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8108917Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8108974Z     return mod(**inputs)
2025-09-07T09:14:29.8109240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8109301Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8109536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8109595Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8109810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8109874Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8110103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8110177Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8110402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8110477Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8110705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8110806Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8110843Z 
2025-09-07T09:14:29.8110932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8111114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8111172Z     return mod(**inputs)
2025-09-07T09:14:29.8111435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8111502Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8111732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8111794Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8112013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8112079Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8112308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8112376Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8112600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8112675Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8112899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:14:29.8112993Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:14:29.8112996Z 
2025-09-07T09:14:29.8113085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8113273Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8113327Z     return mod(**inputs)
2025-09-07T09:14:29.8113555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8113621Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8113852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8113915Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8114127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8114194Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8114480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8114548Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8114811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8114884Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8115113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:14:29.8115203Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:14:29.8115206Z 
2025-09-07T09:14:29.8115295Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8115483Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8115537Z     return mod(**inputs)
2025-09-07T09:14:29.8115776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8115842Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8116073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8116141Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8116352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8116458Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8116689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8116758Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8117022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:14:29.8117097Z     attention_output = self.EncDecAttention(
2025-09-07T09:14:29.8117333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:14:29.8117399Z     attn_output = self.o(attn_output)
2025-09-07T09:14:29.8117404Z 
2025-09-07T09:14:29.8117499Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8117687Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8117745Z     return mod(**inputs)
2025-09-07T09:14:29.8117984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8118049Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8118288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8118352Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8118563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8118638Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8118864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:14:29.8118941Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:14:29.8119170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:14:29.8119297Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:14:29.8119305Z 
2025-09-07T09:14:29.8119378Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8119469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8119664Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8119722Z     return mod(**inputs)
2025-09-07T09:14:29.8120000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8120062Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8120297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8120406Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8120626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8120700Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8120934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8121018Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8121254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:14:29.8121345Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:14:29.8121584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:14:29.8121657Z     return self.weight * hidden_states
2025-09-07T09:14:29.8121662Z 
2025-09-07T09:14:29.8121756Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8121939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8122035Z     return mod(**inputs)
2025-09-07T09:14:29.8122276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8122341Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8122611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8122675Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8122892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8122968Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8123195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8123284Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8123512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8123622Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8123853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:14:29.8123927Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:14:29.8123930Z 
2025-09-07T09:14:29.8124028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8124217Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8124281Z     return mod(**inputs)
2025-09-07T09:14:29.8124514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8124580Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8124814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8124880Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8125098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8125167Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8125394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8125482Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8125750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8125857Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8126119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:14:29.8126194Z     hidden_states = self.act(hidden_states)
2025-09-07T09:14:29.8126200Z 
2025-09-07T09:14:29.8126293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8126478Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8126544Z     return mod(**inputs)
2025-09-07T09:14:29.8126776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:14:29.8126848Z     decoder_outputs = self.decoder(
2025-09-07T09:14:29.8127080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:14:29.8127140Z     layer_outputs = layer_module(
2025-09-07T09:14:29.8127362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:14:29.8127430Z     return super().__call__(*args, **kwargs)
2025-09-07T09:14:29.8127666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:14:29.8127775Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:14:29.8128008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:14:29.8128117Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:14:29.8128372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:14:29.8128450Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:14:29.8128454Z 
2025-09-07T09:14:29.8128526Z cudagraph partition due to non gpu ops
2025-09-07T09:14:29.8128618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8128808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8128864Z     return mod(**inputs)
2025-09-07T09:14:29.8129104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward
2025-09-07T09:14:29.8129216Z     sequence_output = sequence_output * (self.model_dim**-0.5)
2025-09-07T09:14:29.8129219Z 
2025-09-07T09:14:29.8129316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:14:29.8129501Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:14:29.8129559Z     return mod(**inputs)
2025-09-07T09:14:29.8129796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward
2025-09-07T09:14:29.8129873Z     lm_logits = self.lm_head(sequence_output)
2025-09-07T09:14:29.8129876Z 
2025-09-07T09:14:59.0846387Z 
2025-09-07T09:15:00.6954151Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:15:02.3790960Z running benchmark:   3% 1/30 [00:01<00:46,  1.61s/it]
2025-09-07T09:15:03.9760017Z running benchmark:   7% 2/30 [00:03<00:46,  1.65s/it]
2025-09-07T09:15:05.5913078Z running benchmark:  10% 3/30 [00:04<00:43,  1.63s/it]
2025-09-07T09:15:07.2673475Z running benchmark:  13% 4/30 [00:06<00:42,  1.62s/it]
2025-09-07T09:15:08.8770680Z running benchmark:  17% 5/30 [00:08<00:41,  1.64s/it]
2025-09-07T09:15:10.5555148Z running benchmark:  20% 6/30 [00:09<00:39,  1.63s/it]
2025-09-07T09:15:12.1816261Z running benchmark:  23% 7/30 [00:11<00:37,  1.65s/it]
2025-09-07T09:15:13.7804838Z running benchmark:  27% 8/30 [00:13<00:36,  1.64s/it]
2025-09-07T09:15:15.4650918Z running benchmark:  30% 9/30 [00:14<00:34,  1.63s/it]
2025-09-07T09:15:17.0681270Z running benchmark:  33% 10/30 [00:16<00:32,  1.64s/it]
2025-09-07T09:15:18.7486527Z running benchmark:  37% 11/30 [00:17<00:31,  1.63s/it]
2025-09-07T09:15:20.3506556Z running benchmark:  40% 12/30 [00:19<00:29,  1.65s/it]
2025-09-07T09:15:21.9633334Z running benchmark:  43% 13/30 [00:21<00:27,  1.63s/it]
2025-09-07T09:15:23.6406226Z running benchmark:  47% 14/30 [00:22<00:26,  1.63s/it]
2025-09-07T09:15:25.2387559Z running benchmark:  50% 15/30 [00:24<00:24,  1.64s/it]
2025-09-07T09:15:26.8792791Z running benchmark:  53% 16/30 [00:26<00:22,  1.63s/it]
2025-09-07T09:15:28.5005401Z running benchmark:  57% 17/30 [00:27<00:21,  1.63s/it]
2025-09-07T09:15:30.1467593Z running benchmark:  60% 18/30 [00:29<00:19,  1.63s/it]
2025-09-07T09:15:31.7524424Z running benchmark:  63% 19/30 [00:31<00:17,  1.63s/it]
2025-09-07T09:15:33.3681253Z running benchmark:  67% 20/30 [00:32<00:16,  1.63s/it]
2025-09-07T09:15:34.9796594Z running benchmark:  70% 21/30 [00:34<00:14,  1.62s/it]
2025-09-07T09:15:36.5945013Z running benchmark:  73% 22/30 [00:35<00:12,  1.62s/it]
2025-09-07T09:15:38.2136932Z running benchmark:  77% 23/30 [00:37<00:11,  1.62s/it]
2025-09-07T09:15:39.8404526Z running benchmark:  80% 24/30 [00:39<00:09,  1.62s/it]
2025-09-07T09:15:41.4530190Z running benchmark:  83% 25/30 [00:40<00:08,  1.62s/it]
2025-09-07T09:15:43.0656899Z running benchmark:  87% 26/30 [00:42<00:06,  1.62s/it]
2025-09-07T09:15:44.6828379Z running benchmark:  90% 27/30 [00:43<00:04,  1.62s/it]
2025-09-07T09:15:46.3014999Z running benchmark:  93% 28/30 [00:45<00:03,  1.62s/it]
2025-09-07T09:15:47.9164369Z running benchmark:  97% 29/30 [00:47<00:01,  1.62s/it]
2025-09-07T09:15:47.9164894Z running benchmark: 100% 30/30 [00:48<00:00,  1.62s/it]
2025-09-07T09:15:47.9167952Z running benchmark: 100% 30/30 [00:48<00:00,  1.63s/it]
2025-09-07T09:15:48.3212990Z 1.436x
2025-09-07T09:15:48.3217370Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:15:51.8005045Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:15:51.8006177Z   import pynvml  # type: ignore[import]
2025-09-07T09:15:53.8106301Z 
2025-09-07T09:16:03.0660111Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:16:03.0660442Z loading model: 0it [00:09, ?it/s]
2025-09-07T09:16:03.1585002Z cpu  eval  hf_T5_large                        
2025-09-07T09:16:26.8978525Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.8978865Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.8979262Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.8979610Z     return mod(**inputs)
2025-09-07T09:16:26.8980031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.8980426Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.8980800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.8981186Z     layer_outputs = layer_module(
2025-09-07T09:16:26.8981534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.8981893Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.8982279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.8982658Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.8983025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.8983404Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.8983777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward
2025-09-07T09:16:26.8984545Z     position_bias = position_bias + causal_mask
2025-09-07T09:16:26.8984693Z 
2025-09-07T09:16:26.8984798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.8985266Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.8985610Z     return mod(**inputs)
2025-09-07T09:16:26.8985961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.8986340Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.8986698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.8987067Z     layer_outputs = layer_module(
2025-09-07T09:16:26.8987407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.8987771Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.8988131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.8988526Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.8988888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.8989283Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.8989787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.8990170Z     return self.weight * hidden_states
2025-09-07T09:16:26.8990303Z 
2025-09-07T09:16:26.8990412Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.8990838Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.8991167Z     return mod(**inputs)
2025-09-07T09:16:26.8991515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.8991890Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.8992252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.8992695Z     layer_outputs = layer_module(
2025-09-07T09:16:26.8993051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.8993412Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.8993782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.8994161Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.8994527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.8994905Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.8995267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.8995642Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.8995772Z 
2025-09-07T09:16:26.8995876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.8996225Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.8996548Z     return mod(**inputs)
2025-09-07T09:16:26.8996891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.8997258Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.8997650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.8998015Z     layer_outputs = layer_module(
2025-09-07T09:16:26.8998406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.8998766Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.8999171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.8999539Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.8999909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9000281Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9000645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9001012Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9001141Z 
2025-09-07T09:16:26.9001250Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9001603Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9001921Z     return mod(**inputs)
2025-09-07T09:16:26.9002259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9002637Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9002992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9003559Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9003901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9004256Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9004656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9005030Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9005393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9005767Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9006128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9006549Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9006740Z 
2025-09-07T09:16:26.9006836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9007183Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9007498Z     return mod(**inputs)
2025-09-07T09:16:26.9007839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9008214Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9008573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9008947Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9009294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9009651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9010012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9010390Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9010755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9011120Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9011488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9011930Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9012204Z 
2025-09-07T09:16:26.9012304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9012662Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9013033Z     return mod(**inputs)
2025-09-07T09:16:26.9013374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9013740Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9014094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9014461Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9014787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9015148Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9015505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9015879Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9016240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9016603Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9016962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9017373Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9017503Z 
2025-09-07T09:16:26.9017610Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9017947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9018327Z     return mod(**inputs)
2025-09-07T09:16:26.9018708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9019078Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9019433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9019789Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9020120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9020475Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9020837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9021194Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9021563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9021939Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9022296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9022698Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9022856Z 
2025-09-07T09:16:26.9022952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9023289Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9023605Z     return mod(**inputs)
2025-09-07T09:16:26.9023937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9024303Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9024645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9025009Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9025343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9025754Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9026112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9026523Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9026888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9027258Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9027625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9028017Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9028177Z 
2025-09-07T09:16:26.9028269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9028613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9028932Z     return mod(**inputs)
2025-09-07T09:16:26.9029258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9029611Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9029961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9030329Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9030725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9031080Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9031439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9031838Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9032203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9032570Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9032927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9033313Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9033478Z 
2025-09-07T09:16:26.9033576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9033922Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9034235Z     return mod(**inputs)
2025-09-07T09:16:26.9034569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9034934Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9035295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9035661Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9036001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9036373Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9036734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9037134Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9037499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9037867Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9038230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9038612Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9038736Z 
2025-09-07T09:16:26.9038838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9039222Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9039535Z     return mod(**inputs)
2025-09-07T09:16:26.9039869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:26.9040281Z     decoder_outputs = self.decoder(
2025-09-07T09:16:26.9040641Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9041001Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9041340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9041690Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9042064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:26.9042432Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:26.9042786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:26.9043155Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:26.9043526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9043894Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9044023Z 
2025-09-07T09:16:26.9044160Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9044497Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9044823Z     return mod(**inputs)
2025-09-07T09:16:26.9045204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9045586Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9045943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9046313Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9046654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9047011Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9047373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9047739Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9048105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9048477Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9048834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9049205Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9049335Z 
2025-09-07T09:16:26.9049432Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9049774Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9050092Z     return mod(**inputs)
2025-09-07T09:16:26.9050431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9050790Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9051148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9051509Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9051844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9052197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9052553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9052974Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9053330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9053742Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9054110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9054470Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9054610Z 
2025-09-07T09:16:26.9054704Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9055044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9055364Z     return mod(**inputs)
2025-09-07T09:16:26.9055690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9056062Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9056408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9056781Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9057110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9057472Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9057873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9058319Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9058695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9059107Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9059471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9059920Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9060102Z 
2025-09-07T09:16:26.9060214Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9060565Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9060880Z     return mod(**inputs)
2025-09-07T09:16:26.9061219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9061589Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9061943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9062312Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9062642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9063007Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9063368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9063742Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9064128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9064494Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9064872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9065329Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9065535Z 
2025-09-07T09:16:26.9065637Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9065982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9066338Z     return mod(**inputs)
2025-09-07T09:16:26.9066672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9067084Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9067433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9067795Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9068127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9068487Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9068850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9069212Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9069582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9069956Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9070317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9070758Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9070958Z 
2025-09-07T09:16:26.9071052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9071439Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9071759Z     return mod(**inputs)
2025-09-07T09:16:26.9072100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9072458Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9072844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9073223Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9073554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9073912Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9074262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9074631Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9075002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9075376Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9075735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9076106Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9076244Z 
2025-09-07T09:16:26.9076345Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9076705Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9077029Z     return mod(**inputs)
2025-09-07T09:16:26.9077382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9077751Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9078123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9078488Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9078838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9079194Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9079572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9079995Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9080370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9080803Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9081174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9081596Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9081765Z 
2025-09-07T09:16:26.9081863Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9082214Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9082527Z     return mod(**inputs)
2025-09-07T09:16:26.9082874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9083240Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9083605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9083966Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9084310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9084669Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9085073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9085457Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9085823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9086187Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9086589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9086998Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9087155Z 
2025-09-07T09:16:26.9087260Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9087605Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9087922Z     return mod(**inputs)
2025-09-07T09:16:26.9088257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9088631Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9088988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9089345Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9089684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9090048Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9090409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9090787Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9091140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9091514Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9091881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9092275Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9092425Z 
2025-09-07T09:16:26.9092526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9092866Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9093180Z     return mod(**inputs)
2025-09-07T09:16:26.9095160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9095533Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9095886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9096321Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9096659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9097030Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9097395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9097763Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9098180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9098558Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9098939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9099305Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9099437Z 
2025-09-07T09:16:26.9099516Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9099742Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9100088Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9100444Z     return mod(**inputs)
2025-09-07T09:16:26.9100777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9101146Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9101538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9101914Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9102263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9102610Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9102967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9103515Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9103905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9104299Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9104680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9105053Z     return self.weight * hidden_states
2025-09-07T09:16:26.9105194Z 
2025-09-07T09:16:26.9105290Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9105640Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9105951Z     return mod(**inputs)
2025-09-07T09:16:26.9106278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9106650Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9107004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9107377Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9107701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9108059Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9108416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9108801Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9109258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9109660Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9110113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9110491Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9110621Z 
2025-09-07T09:16:26.9110723Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9111060Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9111379Z     return mod(**inputs)
2025-09-07T09:16:26.9111711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9112082Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9112446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9112805Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9113136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9113505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9113872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9114320Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9114713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9115121Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9115576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9115967Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9116100Z 
2025-09-07T09:16:26.9116196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9116541Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9116859Z     return mod(**inputs)
2025-09-07T09:16:26.9117195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9117573Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9117921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9118287Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9118618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9118976Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9119335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9119712Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9120080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9120484Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9120888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9121250Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9121385Z 
2025-09-07T09:16:26.9121468Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9121693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9122037Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9122351Z     return mod(**inputs)
2025-09-07T09:16:26.9122729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9123106Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9123498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9123871Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9124197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9124555Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9124912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9125292Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9125662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9126052Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9126444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9126813Z     return self.weight * hidden_states
2025-09-07T09:16:26.9126938Z 
2025-09-07T09:16:26.9127043Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9127393Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9127739Z     return mod(**inputs)
2025-09-07T09:16:26.9128100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9128484Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9128882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9129261Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9129613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9129973Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9130335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9130717Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9131075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9131453Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9131819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9132194Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9132321Z 
2025-09-07T09:16:26.9132425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9132775Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9133088Z     return mod(**inputs)
2025-09-07T09:16:26.9133427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9133803Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9134152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9134512Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9134854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9135207Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9135568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9135931Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9136352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9136724Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9137092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9137495Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9137627Z 
2025-09-07T09:16:26.9137725Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9138138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9138454Z     return mod(**inputs)
2025-09-07T09:16:26.9138791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9139155Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9139510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9139871Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9140213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9140570Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9140927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9141294Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9141700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9142083Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9142484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9142899Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9143085Z 
2025-09-07T09:16:26.9143179Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9143525Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9143843Z     return mod(**inputs)
2025-09-07T09:16:26.9144169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9144540Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9144892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9145256Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9145588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9145947Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9146313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9146679Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9147045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9147415Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9147779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9148228Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9148450Z 
2025-09-07T09:16:26.9148543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9148883Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9149195Z     return mod(**inputs)
2025-09-07T09:16:26.9149526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9149945Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9150299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9150725Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9151053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9151413Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9151773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9152151Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9152509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9152877Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9153232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9153671Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9153883Z 
2025-09-07T09:16:26.9153987Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9154321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9154638Z     return mod(**inputs)
2025-09-07T09:16:26.9155000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9155376Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9155725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9156091Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9156460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9156831Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9157195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9157573Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9157928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9158300Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9158670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9159043Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9159173Z 
2025-09-07T09:16:26.9159270Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9159616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9159930Z     return mod(**inputs)
2025-09-07T09:16:26.9160269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9160634Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9160984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9161345Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9161686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9162038Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9162392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9162772Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9163143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9163554Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9163922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9164353Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9164526Z 
2025-09-07T09:16:26.9164619Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9164958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9165282Z     return mod(**inputs)
2025-09-07T09:16:26.9165616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9165977Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9166332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9166704Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9167035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9167379Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9167751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9168134Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9168527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9168913Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9169268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9169706Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9190668Z 
2025-09-07T09:16:26.9190837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9191275Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9191622Z     return mod(**inputs)
2025-09-07T09:16:26.9192048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9192474Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9192877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9193293Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9193644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9194016Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9194399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9194789Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9195158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9195532Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9195906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9196351Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9196526Z 
2025-09-07T09:16:26.9196652Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9197026Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9197357Z     return mod(**inputs)
2025-09-07T09:16:26.9197705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9198080Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9198522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9198883Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9199235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9199665Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9200037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9200413Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9200773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9201157Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9201548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9201940Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9202072Z 
2025-09-07T09:16:26.9202189Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9202555Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9202939Z     return mod(**inputs)
2025-09-07T09:16:26.9203510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9203902Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9204329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9204707Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9205058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9205486Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9205862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9206246Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9206628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9207021Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9207408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9207783Z     return self.weight * hidden_states
2025-09-07T09:16:26.9207914Z 
2025-09-07T09:16:26.9208017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9208380Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9208709Z     return mod(**inputs)
2025-09-07T09:16:26.9209060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9209426Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9209794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9210167Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9210510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9210872Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9211233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9211616Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9211991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9212409Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9212865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9213255Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9213436Z 
2025-09-07T09:16:26.9213532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9213882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9214211Z     return mod(**inputs)
2025-09-07T09:16:26.9214545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9214918Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9215274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9215647Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9215979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9216331Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9216693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9217095Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9217481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9217929Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9218411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9218794Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9218928Z 
2025-09-07T09:16:26.9219074Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9219433Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9219754Z     return mod(**inputs)
2025-09-07T09:16:26.9220099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9220472Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9220830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9221208Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9221553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9221911Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9222276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9222654Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9223026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9223422Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9223820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9224192Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9224328Z 
2025-09-07T09:16:26.9224406Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9224648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9224994Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9225316Z     return mod(**inputs)
2025-09-07T09:16:26.9225664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9226045Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9226409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9226839Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9227190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9227580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9227946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9228320Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9228680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9229071Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9229464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9229830Z     return self.weight * hidden_states
2025-09-07T09:16:26.9229961Z 
2025-09-07T09:16:26.9230066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9230416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9230730Z     return mod(**inputs)
2025-09-07T09:16:26.9231064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9231431Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9231819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9232184Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9232514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9232901Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9233270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9233636Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9233992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9234359Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9234713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9235082Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9235208Z 
2025-09-07T09:16:26.9235306Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9235642Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9235947Z     return mod(**inputs)
2025-09-07T09:16:26.9236272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9236629Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9236983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9237342Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9237702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9238064Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9238435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9238809Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9239172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9239538Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9239899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9240300Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9240427Z 
2025-09-07T09:16:26.9240524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9240905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9241219Z     return mod(**inputs)
2025-09-07T09:16:26.9241559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9241931Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9242303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9242662Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9242997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9243374Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9243737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9244123Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9244479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9244845Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9245301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9245725Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9245908Z 
2025-09-07T09:16:26.9246035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9246428Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9246764Z     return mod(**inputs)
2025-09-07T09:16:26.9247115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9247526Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9247875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9248246Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9248579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9248961Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9249319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9249688Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9250051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9250437Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9250792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9251231Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9251434Z 
2025-09-07T09:16:26.9251541Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9251908Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9252222Z     return mod(**inputs)
2025-09-07T09:16:26.9252574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9252934Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9253308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9253719Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9254071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9254434Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9254823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9255185Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9255561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9255931Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9256302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9256731Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9256935Z 
2025-09-07T09:16:26.9257028Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9257382Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9257717Z     return mod(**inputs)
2025-09-07T09:16:26.9258101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9258465Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9258891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9259261Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9259617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9259963Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9260359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9260731Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9261112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9261477Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9261828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9262205Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9262337Z 
2025-09-07T09:16:26.9262434Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9262791Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9263096Z     return mod(**inputs)
2025-09-07T09:16:26.9263440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9263797Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9264155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9264507Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9264849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9265196Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9265566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9265942Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9266298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9266673Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9267036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9267508Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9267666Z 
2025-09-07T09:16:26.9267763Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9268095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9268447Z     return mod(**inputs)
2025-09-07T09:16:26.9268790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9269155Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9269515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9269868Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9270222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9270589Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9270946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9271306Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9271679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9272064Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9272462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9272879Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9273037Z 
2025-09-07T09:16:26.9273129Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9273466Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9273803Z     return mod(**inputs)
2025-09-07T09:16:26.9274159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9274520Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9274862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9275241Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9275567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9275914Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9276290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9276646Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9277007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9277368Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9277746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9278130Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9278292Z 
2025-09-07T09:16:26.9278384Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9278723Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9279039Z     return mod(**inputs)
2025-09-07T09:16:26.9279369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9279740Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9280089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9280464Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9280787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9281196Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9281549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9281970Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9282324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9282708Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9283081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9283437Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9283561Z 
2025-09-07T09:16:26.9283639Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9283861Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9284201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9284504Z     return mod(**inputs)
2025-09-07T09:16:26.9284854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9285215Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9285568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9285972Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9286328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9286684Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9287697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9288092Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9288480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9288865Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9289271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9289638Z     return self.weight * hidden_states
2025-09-07T09:16:26.9289767Z 
2025-09-07T09:16:26.9289885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9290222Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9290532Z     return mod(**inputs)
2025-09-07T09:16:26.9290866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9291225Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9291581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9291939Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9292267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9292641Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9292991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9293381Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9293746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9294164Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9294558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9294937Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9295112Z 
2025-09-07T09:16:26.9295204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9295554Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9295901Z     return mod(**inputs)
2025-09-07T09:16:26.9296223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9296605Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9296953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9297333Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9297653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9298034Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9298412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9298806Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9299191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9299582Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9299980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9300386Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9300515Z 
2025-09-07T09:16:26.9301918Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9302278Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9302638Z     return mod(**inputs)
2025-09-07T09:16:26.9302994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9303554Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9303916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9304284Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9304616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9304990Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9305351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9305728Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9306094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9306501Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9306943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9307306Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9307433Z 
2025-09-07T09:16:26.9307513Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9307739Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9308095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9308410Z     return mod(**inputs)
2025-09-07T09:16:26.9308747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9309127Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9309475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9309838Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9310246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9310598Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9310950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9311411Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9311784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9312184Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9312574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9312937Z     return self.weight * hidden_states
2025-09-07T09:16:26.9313087Z 
2025-09-07T09:16:26.9313186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9313539Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9313854Z     return mod(**inputs)
2025-09-07T09:16:26.9314199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9314572Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9314949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9315312Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9315722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9316078Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9316506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9316885Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9317270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9317645Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9318001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9318385Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9318517Z 
2025-09-07T09:16:26.9318625Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9318972Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9319285Z     return mod(**inputs)
2025-09-07T09:16:26.9319618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9319983Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9320339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9320701Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9321037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9321400Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9321753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9322135Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9322486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9322869Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9323227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9323584Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9323754Z 
2025-09-07T09:16:26.9323862Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9324200Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9324534Z     return mod(**inputs)
2025-09-07T09:16:26.9324899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9325263Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9325612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9325961Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9326302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9326648Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9327021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9327380Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9327743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9328103Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9328464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9328903Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9329086Z 
2025-09-07T09:16:26.9329194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9329534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9329866Z     return mod(**inputs)
2025-09-07T09:16:26.9330230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9330597Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9330955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9331309Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9331650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9331996Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9332348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9332718Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9333068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9333441Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9333792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9334231Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9334443Z 
2025-09-07T09:16:26.9334535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9334893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9335202Z     return mod(**inputs)
2025-09-07T09:16:26.9335527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9335898Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9336239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9336608Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9336933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9337316Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9337677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9338142Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9338523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9338886Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9339261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9339699Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9339903Z 
2025-09-07T09:16:26.9339996Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9340365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9340675Z     return mod(**inputs)
2025-09-07T09:16:26.9341018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9341381Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9341750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9342111Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9342474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9342858Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9343242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9343605Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9344025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9344420Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9344789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9345168Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9345300Z 
2025-09-07T09:16:26.9345399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9345745Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9346078Z     return mod(**inputs)
2025-09-07T09:16:26.9346407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9346763Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9347140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9347493Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9347828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9348198Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9348564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9348927Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9349296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9349662Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9350033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9350428Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9350587Z 
2025-09-07T09:16:26.9350695Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9351080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9351412Z     return mod(**inputs)
2025-09-07T09:16:26.9351745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9352146Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9352514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9352876Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9353211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9353557Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9353934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9354305Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9354686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9355054Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9355429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9355818Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9355972Z 
2025-09-07T09:16:26.9356130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9356486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9356796Z     return mod(**inputs)
2025-09-07T09:16:26.9357170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9357541Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9357892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9358253Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9358577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9358926Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9359274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9359652Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9360004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9360375Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9360732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9361138Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9361296Z 
2025-09-07T09:16:26.9361388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9361725Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9362049Z     return mod(**inputs)
2025-09-07T09:16:26.9362370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9362751Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9363097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9363473Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9363798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9364143Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9364576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9364937Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9365291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9365692Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9366056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9366443Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9366568Z 
2025-09-07T09:16:26.9366664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9366995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9367305Z     return mod(**inputs)
2025-09-07T09:16:26.9367628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9367988Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9368332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9368688Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9369012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9369387Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9369820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9370197Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9370548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9371004Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9371192Z 
2025-09-07T09:16:26.9371270Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9371365Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9371560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9371631Z     return mod(**inputs)
2025-09-07T09:16:26.9371865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9371934Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9372164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9372230Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9372443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9372514Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9372741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9372823Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9373089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9373177Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9373425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9373499Z     return self.weight * hidden_states
2025-09-07T09:16:26.9373502Z 
2025-09-07T09:16:26.9373591Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9373781Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9373840Z     return mod(**inputs)
2025-09-07T09:16:26.9374090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9374194Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9374431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9374537Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9374756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9374831Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9375061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9375152Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9375386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9375496Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9375726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9375798Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9375804Z 
2025-09-07T09:16:26.9375903Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9376088Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9376161Z     return mod(**inputs)
2025-09-07T09:16:26.9376430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9376494Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9376751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9376851Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9377074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9377144Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9377380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9377468Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9377694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9377799Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9378098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9378178Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9378182Z 
2025-09-07T09:16:26.9378281Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9378465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9378525Z     return mod(**inputs)
2025-09-07T09:16:26.9378757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9378842Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9379078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9379140Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9379355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9379430Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9379662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9379742Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9379969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9380137Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9380363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9380473Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9380477Z 
2025-09-07T09:16:26.9380550Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9380643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9380833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9380892Z     return mod(**inputs)
2025-09-07T09:16:26.9381127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9381191Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9381419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9381500Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9381712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9381783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9382012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9382126Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9382360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9382459Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9382730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9382803Z     return self.weight * hidden_states
2025-09-07T09:16:26.9382806Z 
2025-09-07T09:16:26.9382901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9383089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9383152Z     return mod(**inputs)
2025-09-07T09:16:26.9383386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9383448Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9383679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9383743Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9383962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9384033Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9384261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9384342Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9384572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9384647Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9384871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9384942Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9384945Z 
2025-09-07T09:16:26.9385048Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9385232Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9385320Z     return mod(**inputs)
2025-09-07T09:16:26.9385556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9385666Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9385901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9385997Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9386231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9386299Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9386533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9386605Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9386831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9386906Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9387140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9387218Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9387221Z 
2025-09-07T09:16:26.9387309Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9387493Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9387555Z     return mod(**inputs)
2025-09-07T09:16:26.9387824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9387894Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9388124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9388186Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9388445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9388521Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9388756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9388828Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9389059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9389141Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9389373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9389497Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9389501Z 
2025-09-07T09:16:26.9389590Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9389795Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9389856Z     return mod(**inputs)
2025-09-07T09:16:26.9390088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9390163Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9390393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9390459Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9390672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9390740Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9390975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9391047Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9391281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9391399Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9391640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9391820Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9391824Z 
2025-09-07T09:16:26.9391913Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9392105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9392162Z     return mod(**inputs)
2025-09-07T09:16:26.9392399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9392465Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9392698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9392764Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9392975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9393047Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9393274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9393349Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9393613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9393695Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9393942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9394112Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9394115Z 
2025-09-07T09:16:26.9394214Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9394398Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9394459Z     return mod(**inputs)
2025-09-07T09:16:26.9394701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9394766Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9395006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9395075Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9395289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9395362Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9395590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9395664Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9395895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9395975Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9396200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9396272Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9396275Z 
2025-09-07T09:16:26.9396380Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9396567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9396627Z     return mod(**inputs)
2025-09-07T09:16:26.9396863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9396925Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9397200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9397263Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9397519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9397588Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9397823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9397893Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9398122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9398213Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9398442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9398548Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9398551Z 
2025-09-07T09:16:26.9398642Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9398842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9398909Z     return mod(**inputs)
2025-09-07T09:16:26.9399138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9399240Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9399481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9399547Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9399801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9399868Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9400107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9400176Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9400411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9400482Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9400714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9400815Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9400819Z 
2025-09-07T09:16:26.9400909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9401097Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9401158Z     return mod(**inputs)
2025-09-07T09:16:26.9401389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9401457Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9401689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9401768Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9401981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9402048Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9402282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9402353Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9402588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9402658Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9402956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9403188Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9403250Z 
2025-09-07T09:16:26.9403341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9403533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9403591Z     return mod(**inputs)
2025-09-07T09:16:26.9403841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9403903Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9404134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9404203Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9404427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9404499Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9404725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9404800Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9405039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9405178Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9405414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9405486Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9405489Z 
2025-09-07T09:16:26.9405625Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9405720Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9405905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9405969Z     return mod(**inputs)
2025-09-07T09:16:26.9406200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9406268Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9406497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9406562Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9406799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9406869Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9407115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9407198Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9407432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9407519Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9407749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9407830Z     return self.weight * hidden_states
2025-09-07T09:16:26.9407833Z 
2025-09-07T09:16:26.9407925Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9408114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9408172Z     return mod(**inputs)
2025-09-07T09:16:26.9408423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9408502Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9408732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9408860Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9409086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9409217Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9409455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9409539Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9409774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9409884Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9410120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9410192Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9410196Z 
2025-09-07T09:16:26.9410287Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9410481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9410540Z     return mod(**inputs)
2025-09-07T09:16:26.9410777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9410841Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9411467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9411542Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9411760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9411870Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9412104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9412192Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9412427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9412535Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9412768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9412839Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9412843Z 
2025-09-07T09:16:26.9412943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9413132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9413192Z     return mod(**inputs)
2025-09-07T09:16:26.9413427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9413495Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9413734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9413799Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9414016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9414091Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9414323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9414410Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9414644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9414752Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9415029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9415102Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9415138Z 
2025-09-07T09:16:26.9415231Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9415328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9415519Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9415578Z     return mod(**inputs)
2025-09-07T09:16:26.9415808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9415892Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9416121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9416188Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9416400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9416468Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9416714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9416786Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9417065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9417162Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9417394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9417461Z     return self.weight * hidden_states
2025-09-07T09:16:26.9417465Z 
2025-09-07T09:16:26.9417587Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9417778Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9417855Z     return mod(**inputs)
2025-09-07T09:16:26.9418125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9418191Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9418419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9418497Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9418713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9418784Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9419012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9419084Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9419324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9419399Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9419626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9419698Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9419701Z 
2025-09-07T09:16:26.9419796Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9420003Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9420062Z     return mod(**inputs)
2025-09-07T09:16:26.9420296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9420361Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9420608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9420711Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9420922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9421030Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9421258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9421340Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9421571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9421654Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9436747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9436844Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9436850Z 
2025-09-07T09:16:26.9436985Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9437202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9437290Z     return mod(**inputs)
2025-09-07T09:16:26.9437571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9437665Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9437998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9438079Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9438322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9438425Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9438710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9438804Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9439045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9439162Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9439429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9439560Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9439564Z 
2025-09-07T09:16:26.9439673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9439876Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9439940Z     return mod(**inputs)
2025-09-07T09:16:26.9440187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9440266Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9440500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9440564Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9440790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9440864Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9441119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9441208Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9441433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9441525Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9441752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9441961Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9441964Z 
2025-09-07T09:16:26.9442060Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9442297Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9442357Z     return mod(**inputs)
2025-09-07T09:16:26.9442598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9442670Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9442902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9442972Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9443198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9443269Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9443503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9443576Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9443814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9443887Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9444162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9444307Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9444310Z 
2025-09-07T09:16:26.9444400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9444624Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9444698Z     return mod(**inputs)
2025-09-07T09:16:26.9444937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9445006Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9445249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9445321Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9445539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9445627Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9445856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9445928Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9446164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9446254Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9446488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9446560Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9446563Z 
2025-09-07T09:16:26.9446657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9446851Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9446932Z     return mod(**inputs)
2025-09-07T09:16:26.9447170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9447237Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9447476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9447574Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9447788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9447879Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9448158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9448247Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9448475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9448549Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9448791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9448895Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9448898Z 
2025-09-07T09:16:26.9448996Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9449185Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9449243Z     return mod(**inputs)
2025-09-07T09:16:26.9449480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9449562Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9449800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9449894Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9450117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9450188Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9450464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9450544Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9450773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9450851Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9451105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9451199Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9451202Z 
2025-09-07T09:16:26.9451304Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9451488Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9451549Z     return mod(**inputs)
2025-09-07T09:16:26.9451789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9451858Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9452093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9452156Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9452380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9452450Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9452683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9452753Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9452979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9453073Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9453300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9453431Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9453434Z 
2025-09-07T09:16:26.9453526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9453728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9453820Z     return mod(**inputs)
2025-09-07T09:16:26.9454050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9454119Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9454352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9454437Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9454648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9454713Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9454947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9455036Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9455265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9455335Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9455591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9455663Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9455666Z 
2025-09-07T09:16:26.9455754Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9455956Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9456043Z     return mod(**inputs)
2025-09-07T09:16:26.9456275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9456341Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9456576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9456643Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9456853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9456919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9457179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9457274Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9457527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9457653Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9457658Z 
2025-09-07T09:16:26.9457731Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9457825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9458094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9458169Z     return mod(**inputs)
2025-09-07T09:16:26.9458415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9458478Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9458713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9458773Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9458993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9459087Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9459368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9459455Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9459682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9459806Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9460033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9460103Z     return self.weight * hidden_states
2025-09-07T09:16:26.9460106Z 
2025-09-07T09:16:26.9460196Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9460411Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9460476Z     return mod(**inputs)
2025-09-07T09:16:26.9460708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9460801Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9461027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9461093Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9461334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9461411Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9461683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9461766Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9462043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9462157Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9462386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9462462Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9462467Z 
2025-09-07T09:16:26.9462576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9462772Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9462829Z     return mod(**inputs)
2025-09-07T09:16:26.9463066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9463138Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9463385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9463455Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9463671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9463764Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9463995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9464074Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9464307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9464413Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9464646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9464733Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9464737Z 
2025-09-07T09:16:26.9464831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9465022Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9465111Z     return mod(**inputs)
2025-09-07T09:16:26.9465371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9465470Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9465702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9465767Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9465980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9466054Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9466281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9466376Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9466603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9466718Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9466949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9467020Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9467023Z 
2025-09-07T09:16:26.9467099Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9467221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9467425Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9467490Z     return mod(**inputs)
2025-09-07T09:16:26.9467719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9467817Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9468052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9468116Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9468334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9468404Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9468651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9468723Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9468953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9469068Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9469294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9469367Z     return self.weight * hidden_states
2025-09-07T09:16:26.9469370Z 
2025-09-07T09:16:26.9469481Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9469669Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9469728Z     return mod(**inputs)
2025-09-07T09:16:26.9469960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9470040Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9470270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9470337Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9470549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9470623Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9470871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9470997Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9471228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9471332Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9471565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9471634Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9471638Z 
2025-09-07T09:16:26.9471728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9471938Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9471997Z     return mod(**inputs)
2025-09-07T09:16:26.9472230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9472296Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9472522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9472608Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9472819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9472909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9473166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9473241Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9473470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9473572Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9473835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9473907Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9473911Z 
2025-09-07T09:16:26.9474001Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9474184Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9474242Z     return mod(**inputs)
2025-09-07T09:16:26.9474492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9474556Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9474789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9474848Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9475063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9475146Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9475372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9475446Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9475674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9475774Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9476001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9476123Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9476126Z 
2025-09-07T09:16:26.9476220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9476402Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9476496Z     return mod(**inputs)
2025-09-07T09:16:26.9476728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9476808Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9477081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9477147Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9477366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9477432Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9477669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9477769Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9477995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9478070Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9478296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9478447Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9478450Z 
2025-09-07T09:16:26.9478538Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9478767Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9478825Z     return mod(**inputs)
2025-09-07T09:16:26.9479056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9479118Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9479381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9479448Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9479659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9479730Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9479959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9480027Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9480258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9480330Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9480558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9480694Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9480697Z 
2025-09-07T09:16:26.9480792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9480977Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9481032Z     return mod(**inputs)
2025-09-07T09:16:26.9481267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9481330Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9481563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9481623Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9481831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9481901Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9482130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9482243Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9482469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9482576Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9482804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9482870Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9482874Z 
2025-09-07T09:16:26.9482970Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9483162Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9483221Z     return mod(**inputs)
2025-09-07T09:16:26.9483461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9483528Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9483769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9483830Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9484048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9484116Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9484375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9484448Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9484675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9484754Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9485014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9485125Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9485128Z 
2025-09-07T09:16:26.9485220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9485405Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9485467Z     return mod(**inputs)
2025-09-07T09:16:26.9485697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9485765Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9485994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9486055Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9486272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9486340Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9486573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9486644Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9486871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9486947Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9487176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9487271Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9487274Z 
2025-09-07T09:16:26.9487361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9487548Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9487603Z     return mod(**inputs)
2025-09-07T09:16:26.9487830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9487931Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9488165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9488261Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9488476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9488548Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9488774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9488842Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9489073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9489143Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9489371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9489464Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9489469Z 
2025-09-07T09:16:26.9489556Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9489744Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9489798Z     return mod(**inputs)
2025-09-07T09:16:26.9490074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9490136Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9490368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9490479Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9490697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9490772Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9491003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9491078Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9491309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9491382Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9491609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9491680Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9491683Z 
2025-09-07T09:16:26.9491760Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9491853Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9492039Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9492098Z     return mod(**inputs)
2025-09-07T09:16:26.9492331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9492399Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9492627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9492692Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9492910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9492979Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9493208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9493291Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9493568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9493660Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9493925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9494000Z     return self.weight * hidden_states
2025-09-07T09:16:26.9494003Z 
2025-09-07T09:16:26.9494095Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9494284Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9494341Z     return mod(**inputs)
2025-09-07T09:16:26.9494572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9494637Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9494866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9494935Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9495146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9495214Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9495443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9495554Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9495793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9495909Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9496174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9496247Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9496250Z 
2025-09-07T09:16:26.9496341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9496529Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9496586Z     return mod(**inputs)
2025-09-07T09:16:26.9496819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9496883Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9497116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9497182Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9497393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9497462Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9497689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9497766Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9498056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9498160Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9498391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9498462Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9498465Z 
2025-09-07T09:16:26.9498558Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9498743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9498802Z     return mod(**inputs)
2025-09-07T09:16:26.9499045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9499144Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9499376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9499599Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9499812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9499883Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9500110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9500188Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9500416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9500514Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9500743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9500810Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9500815Z 
2025-09-07T09:16:26.9500893Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9500983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9501172Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9501264Z     return mod(**inputs)
2025-09-07T09:16:26.9501499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9501565Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9501829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9501894Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9502109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9502174Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9502404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9502477Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9502709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9502808Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9503034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9503286Z     return self.weight * hidden_states
2025-09-07T09:16:26.9503289Z 
2025-09-07T09:16:26.9503383Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9503567Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9503624Z     return mod(**inputs)
2025-09-07T09:16:26.9503856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9503919Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9504148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9504217Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9504428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9504496Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9504723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9504794Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9505119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9505194Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9505422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9505542Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9505545Z 
2025-09-07T09:16:26.9505642Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9505832Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9505888Z     return mod(**inputs)
2025-09-07T09:16:26.9506122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9506185Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9506414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9506477Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9506687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9506758Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9506983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9507056Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9507331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9507402Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9507678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9507748Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9507752Z 
2025-09-07T09:16:26.9507842Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9508023Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9508083Z     return mod(**inputs)
2025-09-07T09:16:26.9508313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9508374Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9508605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9508664Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9508877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9508943Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9509169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9509241Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9509466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9509538Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9509763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9509883Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9509889Z 
2025-09-07T09:16:26.9509977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9510157Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9510215Z     return mod(**inputs)
2025-09-07T09:16:26.9510446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9510544Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9510774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9510868Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9511081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9511145Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9511375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9511442Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9511665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9511739Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9511963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9512114Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9512118Z 
2025-09-07T09:16:26.9512204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9512389Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9512444Z     return mod(**inputs)
2025-09-07T09:16:26.9512718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9512784Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9513012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9513073Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9513316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9513384Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9513612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9513682Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9513912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9513983Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9514208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9514345Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9514348Z 
2025-09-07T09:16:26.9514434Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9514618Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9514675Z     return mod(**inputs)
2025-09-07T09:16:26.9514905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9514965Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9515198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9515262Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9515473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9515542Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9515767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9515833Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9516064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9516202Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9516430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9516531Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9516534Z 
2025-09-07T09:16:26.9516624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9516808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9516863Z     return mod(**inputs)
2025-09-07T09:16:26.9517094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9517155Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9517389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9517449Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9517660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9517729Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9517957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9518028Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9518286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9518357Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9518584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9518712Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9518716Z 
2025-09-07T09:16:26.9518807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9518988Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9519049Z     return mod(**inputs)
2025-09-07T09:16:26.9519279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9519342Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9519579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9519640Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9519856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9519921Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9520147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9520217Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9520442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9520514Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9520739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9520830Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9520835Z 
2025-09-07T09:16:26.9520924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9521105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9521166Z     return mod(**inputs)
2025-09-07T09:16:26.9521396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9521464Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9521729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9521790Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9522004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9522102Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9522330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9522401Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9522628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9522700Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9522926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9523024Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9523027Z 
2025-09-07T09:16:26.9523113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9523295Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9523353Z     return mod(**inputs)
2025-09-07T09:16:26.9523584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9523682Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9523913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9523974Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9524215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9524281Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9524510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9524577Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9524803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9524871Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9525095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9525163Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9525167Z 
2025-09-07T09:16:26.9525254Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9525436Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9525491Z     return mod(**inputs)
2025-09-07T09:16:26.9525722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9525783Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9526010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9526075Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9526286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9526354Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9526582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9526649Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9526879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9527002Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9527040Z 
2025-09-07T09:16:26.9527113Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9527204Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9527387Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9527481Z     return mod(**inputs)
2025-09-07T09:16:26.9527711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9527779Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9528008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9528070Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9528281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9528347Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9528579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9528660Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9528887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9528974Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9529234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9529306Z     return self.weight * hidden_states
2025-09-07T09:16:26.9529308Z 
2025-09-07T09:16:26.9529394Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9529581Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9529668Z     return mod(**inputs)
2025-09-07T09:16:26.9529901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9529968Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9530198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9530264Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9530473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9530541Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9530768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9530845Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9531076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9531179Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9531409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9531478Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9531482Z 
2025-09-07T09:16:26.9531570Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9531764Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9531818Z     return mod(**inputs)
2025-09-07T09:16:26.9532051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9532113Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9532345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9532406Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9532617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9532721Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9532946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9533072Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9533301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9533403Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9533634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9533706Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9533709Z 
2025-09-07T09:16:26.9533801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9533982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9534039Z     return mod(**inputs)
2025-09-07T09:16:26.9534269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9534332Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9534565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9534625Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9534869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9534935Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9535160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9535268Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9535496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9535601Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9535826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9535895Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9535897Z 
2025-09-07T09:16:26.9535972Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9536059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9536244Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9536298Z     return mod(**inputs)
2025-09-07T09:16:26.9536529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9536595Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9536825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9536891Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9537100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9537171Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9537397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9537467Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9537696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9537795Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9538059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9538162Z     return self.weight * hidden_states
2025-09-07T09:16:26.9538165Z 
2025-09-07T09:16:26.9538253Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9538437Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9538528Z     return mod(**inputs)
2025-09-07T09:16:26.9538760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9538822Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9539054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9539118Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9539327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9539396Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9539621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9539694Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9539918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9539988Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9540216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9540316Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9540319Z 
2025-09-07T09:16:26.9540411Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9540593Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9540650Z     return mod(**inputs)
2025-09-07T09:16:26.9540912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9540977Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9541208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9541270Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9541479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9541546Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9541772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9541843Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9542067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9542140Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9542364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9542432Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9542435Z 
2025-09-07T09:16:26.9542528Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9542709Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9542770Z     return mod(**inputs)
2025-09-07T09:16:26.9543003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9543065Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9543299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9543362Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9543575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9543679Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9543913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9544016Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9544241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9544317Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9544544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9544664Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9544667Z 
2025-09-07T09:16:26.9544755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9544939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9545008Z     return mod(**inputs)
2025-09-07T09:16:26.9545247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9545320Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9545556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9545620Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9545879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9545953Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9546191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9546264Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9546527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9546602Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9546830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9546983Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9546986Z 
2025-09-07T09:16:26.9547083Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9547279Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9547337Z     return mod(**inputs)
2025-09-07T09:16:26.9547568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9547634Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9547864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9547930Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9548150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9548217Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9548447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9548516Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9548748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9548819Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9549048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9549186Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9549189Z 
2025-09-07T09:16:26.9549313Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9549504Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9549560Z     return mod(**inputs)
2025-09-07T09:16:26.9549830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9549895Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9550127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9550191Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9550404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9550473Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9550700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9550771Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9550997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9551069Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9551294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9551364Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9551367Z 
2025-09-07T09:16:26.9551492Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9551677Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9551732Z     return mod(**inputs)
2025-09-07T09:16:26.9551997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9552061Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9552297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9552358Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9552571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9552640Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9552867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9552939Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9553164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9553235Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9553461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9553566Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9553570Z 
2025-09-07T09:16:26.9553660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9553843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9553903Z     return mod(**inputs)
2025-09-07T09:16:26.9554131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9554195Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9554427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9554487Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9554704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9554769Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9555074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9555145Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9555432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9555507Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9555734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9555834Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9555837Z 
2025-09-07T09:16:26.9555924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9556109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9556167Z     return mod(**inputs)
2025-09-07T09:16:26.9556399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9556466Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9556692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9556754Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9556969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9557070Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9557301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9557369Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9557633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9557702Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9557932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9558026Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9558031Z 
2025-09-07T09:16:26.9558118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9558304Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9558360Z     return mod(**inputs)
2025-09-07T09:16:26.9558589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9558654Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9558882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9558945Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9559158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9559225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9559459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9559528Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9559754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9559822Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9560057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9560125Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9560128Z 
2025-09-07T09:16:26.9560197Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9560288Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9560516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9560579Z     return mod(**inputs)
2025-09-07T09:16:26.9560807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9560904Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9561137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9561199Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9561412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9561477Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9561703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9561789Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9562014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9562099Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9562326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9562395Z     return self.weight * hidden_states
2025-09-07T09:16:26.9562398Z 
2025-09-07T09:16:26.9562519Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9562703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9562760Z     return mod(**inputs)
2025-09-07T09:16:26.9562990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9563084Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9563317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9563381Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9563607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9563674Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9563903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9563980Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9564203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9564311Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9564538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9564610Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9564613Z 
2025-09-07T09:16:26.9564705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9564890Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9564949Z     return mod(**inputs)
2025-09-07T09:16:26.9565178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9565242Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9565472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9565543Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9565753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9565820Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9566053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9566164Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9566400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9566539Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9566772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9566843Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9566846Z 
2025-09-07T09:16:26.9566937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9567126Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9567186Z     return mod(**inputs)
2025-09-07T09:16:26.9567423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9567487Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9567716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9567783Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9567998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9568069Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9568328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9568409Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9568673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9568779Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9569015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9569085Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9569090Z 
2025-09-07T09:16:26.9569164Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9569253Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9569438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9569502Z     return mod(**inputs)
2025-09-07T09:16:26.9569734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9569802Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9570033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9570096Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9570317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9570384Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9570615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9570690Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9570916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9571020Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9571245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9571319Z     return self.weight * hidden_states
2025-09-07T09:16:26.9571322Z 
2025-09-07T09:16:26.9571409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9571644Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9571701Z     return mod(**inputs)
2025-09-07T09:16:26.9571932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9572029Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9572259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9572326Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9572541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9572608Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9572839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9572911Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9573150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9573220Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9573451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9573523Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9573526Z 
2025-09-07T09:16:26.9573615Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9573836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9573893Z     return mod(**inputs)
2025-09-07T09:16:26.9574127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9574234Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9574468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9574537Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9574747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9574819Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9575042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9575109Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9575339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9575408Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9575636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9575704Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9575708Z 
2025-09-07T09:16:26.9575798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9575981Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9576038Z     return mod(**inputs)
2025-09-07T09:16:26.9576271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9576333Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9576566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9576625Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9576835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9576905Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9577133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9577238Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9577464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9577566Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9577795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9577914Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9577918Z 
2025-09-07T09:16:26.9578042Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9578232Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9578289Z     return mod(**inputs)
2025-09-07T09:16:26.9578531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9578593Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9578826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9578888Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9579106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9579172Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9579434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9579508Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9579736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9579839Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9580068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9580216Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9580222Z 
2025-09-07T09:16:26.9580315Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9580503Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9580562Z     return mod(**inputs)
2025-09-07T09:16:26.9580793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9580860Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9581089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9581151Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9581366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9581433Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9581661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9581731Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9581957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9582029Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9582255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9582395Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9582398Z 
2025-09-07T09:16:26.9582487Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9582674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9582763Z     return mod(**inputs)
2025-09-07T09:16:26.9582992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9583060Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9583326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9583395Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9583613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9583680Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9583909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9583977Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9584207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9584278Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9584503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9584584Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9584587Z 
2025-09-07T09:16:26.9584702Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9584921Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9584985Z     return mod(**inputs)
2025-09-07T09:16:26.9585227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9585289Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9585553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9585635Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9585850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9585923Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9586155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9586240Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9586478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9586547Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9586793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9586896Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9586899Z 
2025-09-07T09:16:26.9586996Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9587183Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9587241Z     return mod(**inputs)
2025-09-07T09:16:26.9587476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9587557Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9587793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9587856Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9588067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9588158Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9588389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9588498Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9588725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9588795Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9589072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9589171Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9589174Z 
2025-09-07T09:16:26.9589271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9589455Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9589516Z     return mod(**inputs)
2025-09-07T09:16:26.9589750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9589814Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9590048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9590107Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9590322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9590406Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9590639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9590746Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9590976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9591053Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9591314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9591423Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9591432Z 
2025-09-07T09:16:26.9591520Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9591715Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9591775Z     return mod(**inputs)
2025-09-07T09:16:26.9592005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9592075Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9592305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9592365Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9592582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9592650Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9592884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9592953Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9593179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9593275Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9593505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9593598Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9593602Z 
2025-09-07T09:16:26.9593687Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9593872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9593931Z     return mod(**inputs)
2025-09-07T09:16:26.9594161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9594267Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9594505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9594606Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9594818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9594885Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9595130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9595196Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9595435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9595560Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9595565Z 
2025-09-07T09:16:26.9595640Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9595747Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9595936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9596001Z     return mod(**inputs)
2025-09-07T09:16:26.9596240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9596369Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9596605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9596666Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9596923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9596993Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9597229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9597315Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9597542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9597652Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9597884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9597972Z     return self.weight * hidden_states
2025-09-07T09:16:26.9597975Z 
2025-09-07T09:16:26.9598064Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9598252Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9598315Z     return mod(**inputs)
2025-09-07T09:16:26.9598546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9598630Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9598860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9598927Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9599137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9599208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9599454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9599533Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9599764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9599871Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9600161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9600241Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9600277Z 
2025-09-07T09:16:26.9600369Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9600559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9600614Z     return mod(**inputs)
2025-09-07T09:16:26.9600863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9600932Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9601164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9601230Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9601466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9601538Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9601765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9601841Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9602090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9602224Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9602459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9602529Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9602533Z 
2025-09-07T09:16:26.9602673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9602887Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9602944Z     return mod(**inputs)
2025-09-07T09:16:26.9603330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9603395Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9603630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9603711Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9603922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9603994Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9604221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9604300Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9604525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9604626Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9604876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9604949Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9604952Z 
2025-09-07T09:16:26.9605037Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9605128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9605313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9605373Z     return mod(**inputs)
2025-09-07T09:16:26.9605613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9605681Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9605981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9606070Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9606288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9606426Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9606669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9606745Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9606977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9607078Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9607323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9607399Z     return self.weight * hidden_states
2025-09-07T09:16:26.9607402Z 
2025-09-07T09:16:26.9607492Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9607680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9607740Z     return mod(**inputs)
2025-09-07T09:16:26.9607969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9608038Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9608311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9608380Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9608596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9608718Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9608951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9609022Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9609256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9609332Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9609583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9609656Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9609659Z 
2025-09-07T09:16:26.9609754Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9609947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9610004Z     return mod(**inputs)
2025-09-07T09:16:26.9610243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9610308Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9610539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9610605Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9610820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9610898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9611127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9611203Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9611430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9611501Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9611732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9611841Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9611844Z 
2025-09-07T09:16:26.9611943Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9612160Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9612230Z     return mod(**inputs)
2025-09-07T09:16:26.9612473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9612537Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9612772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9612836Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9613064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9613133Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9613360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9613437Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9613667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9613741Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9613999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9614122Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9614125Z 
2025-09-07T09:16:26.9614218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9614432Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9614518Z     return mod(**inputs)
2025-09-07T09:16:26.9614752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9614816Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9615053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9615114Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9615351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9615417Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9615649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9615720Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9615945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9616036Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9616264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9616416Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9616420Z 
2025-09-07T09:16:26.9616507Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9616695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9616754Z     return mod(**inputs)
2025-09-07T09:16:26.9616981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9617046Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9617279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9617388Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9617610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9617675Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9618016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9618107Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9618343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9618412Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9618639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9618784Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9618788Z 
2025-09-07T09:16:26.9618879Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9619087Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9619143Z     return mod(**inputs)
2025-09-07T09:16:26.9619376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9619441Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9619706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9619783Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9619997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9620066Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9620324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9620395Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9620637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9620709Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9620939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9621008Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9621011Z 
2025-09-07T09:16:26.9621103Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9621292Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9621347Z     return mod(**inputs)
2025-09-07T09:16:26.9621583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9621660Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9621888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9621955Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9622167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9622238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9622486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9622560Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9622787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9622857Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9623099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9623232Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9623235Z 
2025-09-07T09:16:26.9623328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9623546Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9623600Z     return mod(**inputs)
2025-09-07T09:16:26.9623836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9623901Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9624145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9624207Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9624420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9624493Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9624719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9624816Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9625041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9625114Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9625371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9625464Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9625467Z 
2025-09-07T09:16:26.9625559Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9625775Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9625868Z     return mod(**inputs)
2025-09-07T09:16:26.9626098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9626162Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9626397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9626482Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9626695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9626762Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9627012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9627081Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9627306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9627378Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9627605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9627737Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9627742Z 
2025-09-07T09:16:26.9627833Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9628013Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9628069Z     return mod(**inputs)
2025-09-07T09:16:26.9628300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9628375Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9628604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9628665Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9628879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9628980Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9629212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9629314Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9629571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9629647Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9629877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9629947Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9629950Z 
2025-09-07T09:16:26.9630023Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9630119Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9630311Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9630368Z     return mod(**inputs)
2025-09-07T09:16:26.9630604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9630671Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9630904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9630999Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9631218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9631292Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9631551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9631637Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9631865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9631955Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9632181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9632248Z     return self.weight * hidden_states
2025-09-07T09:16:26.9632252Z 
2025-09-07T09:16:26.9632347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9632533Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9632597Z     return mod(**inputs)
2025-09-07T09:16:26.9632833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9632899Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9633132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9633197Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9633415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9633492Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9633726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9633817Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9634044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9634159Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9634387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9634504Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9634507Z 
2025-09-07T09:16:26.9634601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9634795Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9634891Z     return mod(**inputs)
2025-09-07T09:16:26.9635123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9635193Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9635423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9635486Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9635703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9635774Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9636011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9636091Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9636322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9636426Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9636683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9636760Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9636763Z 
2025-09-07T09:16:26.9636852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9637040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9637129Z     return mod(**inputs)
2025-09-07T09:16:26.9637366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9637436Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9637665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9637735Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9637946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9638015Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9638250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9638326Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9638563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9638665Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9638906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9638980Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9638985Z 
2025-09-07T09:16:26.9639061Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9639157Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9639343Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9639409Z     return mod(**inputs)
2025-09-07T09:16:26.9639643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9639709Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9639950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9640015Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9640294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9640365Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9640594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9640710Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9640942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9641050Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9641278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9641353Z     return self.weight * hidden_states
2025-09-07T09:16:26.9641356Z 
2025-09-07T09:16:26.9641448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9641634Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9641698Z     return mod(**inputs)
2025-09-07T09:16:26.9641927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9642000Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9642230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9642330Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9642555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9642624Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9642904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9642977Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9643216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9643290Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9643516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9643595Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9643598Z 
2025-09-07T09:16:26.9643689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9643882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9643940Z     return mod(**inputs)
2025-09-07T09:16:26.9644170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9644243Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9644473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9644546Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9644760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9644827Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9645064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9645136Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9645372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9645444Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9645678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9645748Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9645803Z 
2025-09-07T09:16:26.9645893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9646084Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9646188Z     return mod(**inputs)
2025-09-07T09:16:26.9646428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9646493Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9646725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9646796Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9647012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9647087Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9647317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9647389Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9647623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9647696Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9647933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9648088Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9648091Z 
2025-09-07T09:16:26.9648187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9648371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9648431Z     return mod(**inputs)
2025-09-07T09:16:26.9648704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9648770Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9649010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9649075Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9649288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9649363Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9649596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9649677Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9649903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9649977Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9650214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9650362Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9650365Z 
2025-09-07T09:16:26.9650463Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9650646Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9650710Z     return mod(**inputs)
2025-09-07T09:16:26.9650942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9651008Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9651243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9651305Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9651528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9651631Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9651861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9659299Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9659605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9659692Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9659970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9660125Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9660129Z 
2025-09-07T09:16:26.9660238Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9660443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9660514Z     return mod(**inputs)
2025-09-07T09:16:26.9660768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9660844Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9661092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9661158Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9661451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9661528Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9661768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9661882Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9662117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9662202Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9662430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9662515Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9662521Z 
2025-09-07T09:16:26.9662625Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9662829Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9662894Z     return mod(**inputs)
2025-09-07T09:16:26.9663130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9663201Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9663435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9663501Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9663723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9663795Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9664030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9664101Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9664328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9664403Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9664627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9664739Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9664743Z 
2025-09-07T09:16:26.9664836Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9665069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9665129Z     return mod(**inputs)
2025-09-07T09:16:26.9665364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9665466Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9665699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9665768Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9665982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9666047Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9666279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9666349Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9666581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9666652Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9666880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9666979Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9666983Z 
2025-09-07T09:16:26.9667106Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9667294Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9667350Z     return mod(**inputs)
2025-09-07T09:16:26.9667636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9667702Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9667931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9667997Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9668212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9668282Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9668509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9668579Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9668809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9668879Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9669113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9669209Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9669212Z 
2025-09-07T09:16:26.9669309Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9669494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9669555Z     return mod(**inputs)
2025-09-07T09:16:26.9669793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9669857Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9670095Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9670156Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9670374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9670449Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9670713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9670786Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9671011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9671113Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9671347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9671416Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9671420Z 
2025-09-07T09:16:26.9671514Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9671695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9671758Z     return mod(**inputs)
2025-09-07T09:16:26.9671987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9672052Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9672285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9672348Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9672563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9672628Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9672887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9672963Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9673223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9673353Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9673358Z 
2025-09-07T09:16:26.9673431Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9673522Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9673708Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9673768Z     return mod(**inputs)
2025-09-07T09:16:26.9674004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9674066Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9674298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9674358Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9674573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9674644Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9674874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9674960Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9675208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9675294Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9675529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9675597Z     return self.weight * hidden_states
2025-09-07T09:16:26.9675601Z 
2025-09-07T09:16:26.9675693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9675877Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9675936Z     return mod(**inputs)
2025-09-07T09:16:26.9676171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9676272Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9676506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9676602Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9676816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9676890Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9677119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9677206Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9677435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9677548Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9677778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9677847Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9677851Z 
2025-09-07T09:16:26.9677947Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9678130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9678189Z     return mod(**inputs)
2025-09-07T09:16:26.9678457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9678528Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9678775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9678870Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9679093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9679164Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9679395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9679482Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9679708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9679817Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9680043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9680116Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9680122Z 
2025-09-07T09:16:26.9680218Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9680409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9680473Z     return mod(**inputs)
2025-09-07T09:16:26.9680708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9680777Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9681006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9681069Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9681289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9681356Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9681587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9681661Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9681888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9682026Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9682255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9682359Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9682363Z 
2025-09-07T09:16:26.9682436Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9682534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9682721Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9682779Z     return mod(**inputs)
2025-09-07T09:16:26.9683013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9683076Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9683313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9683374Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9683587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9683663Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9683891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9684000Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9684230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9684330Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9684595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9684668Z     return self.weight * hidden_states
2025-09-07T09:16:26.9684671Z 
2025-09-07T09:16:26.9684768Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9684952Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9685017Z     return mod(**inputs)
2025-09-07T09:16:26.9685261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9685327Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9685576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9685638Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9685858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9685927Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9686159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9686234Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9686463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9686546Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9686771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9686845Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9686848Z 
2025-09-07T09:16:26.9686946Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9687139Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9687208Z     return mod(**inputs)
2025-09-07T09:16:26.9687440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9687542Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9687773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9687867Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9688089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9688155Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9688388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9688457Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9688683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9688764Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9688989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9689063Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9689066Z 
2025-09-07T09:16:26.9689156Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9689346Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9689406Z     return mod(**inputs)
2025-09-07T09:16:26.9689682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9689754Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9689986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9690055Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9690299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9690368Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9690601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9690674Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9690906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9690976Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9691203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9691331Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9691334Z 
2025-09-07T09:16:26.9691423Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9691614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9691671Z     return mod(**inputs)
2025-09-07T09:16:26.9691900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9691966Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9692198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9692265Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9692478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9692552Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9692778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9692847Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9693079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9693182Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9693413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9693603Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9693607Z 
2025-09-07T09:16:26.9693697Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9693890Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9693947Z     return mod(**inputs)
2025-09-07T09:16:26.9694182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9694244Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9694482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9694543Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9694755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9694831Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9695060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9695133Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9695391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9695463Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9695697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9695869Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9695873Z 
2025-09-07T09:16:26.9695967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9696151Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9696208Z     return mod(**inputs)
2025-09-07T09:16:26.9696443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9696505Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9696746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9696808Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9697025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9697092Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9697321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9697397Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9697622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9697700Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9697924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9698036Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9698039Z 
2025-09-07T09:16:26.9698134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9698318Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9698379Z     return mod(**inputs)
2025-09-07T09:16:26.9698612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9698675Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9698950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9699012Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9699265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9699331Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9699564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9699634Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9699860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9699937Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9700171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9700272Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9700275Z 
2025-09-07T09:16:26.9700365Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9700553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9700610Z     return mod(**inputs)
2025-09-07T09:16:26.9700840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9700962Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9701197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9701262Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9701509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9701580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9701808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9701880Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9702107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9702179Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9702408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9702499Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9702503Z 
2025-09-07T09:16:26.9702594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9702778Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9702834Z     return mod(**inputs)
2025-09-07T09:16:26.9703234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9703299Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9703537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9703599Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9703812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9703880Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9704118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9704192Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9704419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9704493Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9704806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9704908Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9704954Z 
2025-09-07T09:16:26.9705052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9705245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9705304Z     return mod(**inputs)
2025-09-07T09:16:26.9705540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9705606Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9705841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9705904Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9706124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9706195Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9706429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9706501Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9706732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9706852Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9707081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9707151Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9707155Z 
2025-09-07T09:16:26.9707270Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9707364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9707559Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9707616Z     return mod(**inputs)
2025-09-07T09:16:26.9707851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9707916Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9708146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9708210Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9708422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9708492Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9708718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9708801Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9709030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9709114Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9709343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9709410Z     return self.weight * hidden_states
2025-09-07T09:16:26.9709414Z 
2025-09-07T09:16:26.9709508Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9709690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9709744Z     return mod(**inputs)
2025-09-07T09:16:26.9709975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9710039Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9710268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9710367Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9710588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9710711Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9710938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9711023Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9711249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9711359Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9711586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9711658Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9711661Z 
2025-09-07T09:16:26.9711752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9711936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9711995Z     return mod(**inputs)
2025-09-07T09:16:26.9712226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9712288Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9712552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9712613Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9712828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9712924Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9713154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9713237Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9713463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9713571Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9713802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9713874Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9713877Z 
2025-09-07T09:16:26.9713967Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9714152Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9714213Z     return mod(**inputs)
2025-09-07T09:16:26.9714443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9714512Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9714741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9714805Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9715022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9715089Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9715320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9715396Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9715626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9715725Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9715989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9716065Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9716099Z 
2025-09-07T09:16:26.9716173Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9716265Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9716449Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9716506Z     return mod(**inputs)
2025-09-07T09:16:26.9716736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9716798Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9717035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9717094Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9717306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9717376Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9717604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9717678Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9717933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9718033Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9718262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9718327Z     return self.weight * hidden_states
2025-09-07T09:16:26.9718330Z 
2025-09-07T09:16:26.9718456Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9718646Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9718703Z     return mod(**inputs)
2025-09-07T09:16:26.9718935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9718999Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9719233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9719295Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9719512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9719578Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9719804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9719878Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9720105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9720178Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9720404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9720475Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9720478Z 
2025-09-07T09:16:26.9720569Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9720751Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9720811Z     return mod(**inputs)
2025-09-07T09:16:26.9721038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9721104Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9721335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9721432Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9721645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9721745Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9721974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9722045Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9722273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9722348Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9722575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9722648Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9722653Z 
2025-09-07T09:16:26.9722740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9722923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9722979Z     return mod(**inputs)
2025-09-07T09:16:26.9723208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9723273Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9723746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9723808Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9724021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9724086Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9724346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9724419Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9724646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9724718Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9724946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9725069Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9725073Z 
2025-09-07T09:16:26.9725162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9725345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9725400Z     return mod(**inputs)
2025-09-07T09:16:26.9725630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9725693Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9725919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9725983Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9726193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9726260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9726487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9726555Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9726784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9726853Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9727086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9727267Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9727270Z 
2025-09-07T09:16:26.9727360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9727577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9727632Z     return mod(**inputs)
2025-09-07T09:16:26.9727863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9727924Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9728152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9728212Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9728423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9728493Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9728720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9728793Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9729017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9729087Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9729345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9729482Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9729485Z 
2025-09-07T09:16:26.9729574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9729790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9729849Z     return mod(**inputs)
2025-09-07T09:16:26.9730079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9730145Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9730376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9730437Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9730658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9730723Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9730947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9731021Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9731245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9731321Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9731547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9731623Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9731627Z 
2025-09-07T09:16:26.9731713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9731896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9731953Z     return mod(**inputs)
2025-09-07T09:16:26.9732181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9732243Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9732471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9732579Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9732791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9732856Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9733120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9733187Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9733414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9733484Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9733710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9733811Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9733815Z 
2025-09-07T09:16:26.9733904Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9734088Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9734142Z     return mod(**inputs)
2025-09-07T09:16:26.9734371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9734437Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9734697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9734761Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9734970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9735037Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9735296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9735367Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9735598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9735668Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9735898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9735991Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9735994Z 
2025-09-07T09:16:26.9736084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9736270Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9736327Z     return mod(**inputs)
2025-09-07T09:16:26.9736556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9736619Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9736849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9736913Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9737125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9737199Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9737427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9737497Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9737723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9737791Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9738139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9738261Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9738264Z 
2025-09-07T09:16:26.9738352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9738535Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9738624Z     return mod(**inputs)
2025-09-07T09:16:26.9738859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9738921Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9739155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9739214Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9739429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9739495Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9739720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9739791Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9740020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9740088Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9740347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9740417Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9740420Z 
2025-09-07T09:16:26.9740510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9740691Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9740779Z     return mod(**inputs)
2025-09-07T09:16:26.9741013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9741078Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9741309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9741370Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9741585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9741651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9741881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9741951Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9742177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9742306Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9742311Z 
2025-09-07T09:16:26.9742382Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9742473Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9742655Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9742713Z     return mod(**inputs)
2025-09-07T09:16:26.9742945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9743007Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9743243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9743304Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9743516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9743583Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9743847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9743931Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9744195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9744280Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9744510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9744581Z     return self.weight * hidden_states
2025-09-07T09:16:26.9744584Z 
2025-09-07T09:16:26.9744674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9744860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9744919Z     return mod(**inputs)
2025-09-07T09:16:26.9745147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9745211Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9745441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9745503Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9745717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9745821Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9746052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9746132Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9746393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9746504Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9746738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9746807Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9746814Z 
2025-09-07T09:16:26.9746904Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9747086Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9747145Z     return mod(**inputs)
2025-09-07T09:16:26.9747378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9747442Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9747672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9747733Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9747945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9748011Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9748239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9748318Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9748543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9748651Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9748876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9748948Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9748951Z 
2025-09-07T09:16:26.9749039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9749223Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9749330Z     return mod(**inputs)
2025-09-07T09:16:26.9749560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9749657Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9749886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9749951Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9750164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9750230Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9750457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9750532Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9750763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9750864Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9751092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9751169Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9751172Z 
2025-09-07T09:16:26.9751241Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9751391Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9751578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9751637Z     return mod(**inputs)
2025-09-07T09:16:26.9751926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9751988Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9752220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9752280Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9752494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9752560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9752786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9752861Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9753088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9753197Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9753432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9753502Z     return self.weight * hidden_states
2025-09-07T09:16:26.9753511Z 
2025-09-07T09:16:26.9753603Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9753785Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9753848Z     return mod(**inputs)
2025-09-07T09:16:26.9754078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9754144Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9754375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9754437Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9754654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9754720Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9754995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9755067Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9755297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9755408Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9755637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9755713Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9755716Z 
2025-09-07T09:16:26.9755805Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9755990Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9756050Z     return mod(**inputs)
2025-09-07T09:16:26.9756282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9756352Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9756582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9756650Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9756861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9756927Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9757190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9757261Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9757491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9757594Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9757824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9757894Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9757898Z 
2025-09-07T09:16:26.9757985Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9758174Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9758230Z     return mod(**inputs)
2025-09-07T09:16:26.9758464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9758531Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9758758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9758823Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9759032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9759101Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9759326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9759394Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9759625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9759694Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9759925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9760045Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9760048Z 
2025-09-07T09:16:26.9760134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9760321Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9760420Z     return mod(**inputs)
2025-09-07T09:16:26.9760653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9760716Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9760985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9761047Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9761261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9761334Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9761559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9761632Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9761861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9761931Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9762161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9762304Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9762308Z 
2025-09-07T09:16:26.9762401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9762620Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9762676Z     return mod(**inputs)
2025-09-07T09:16:26.9762916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9762976Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9763243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9763307Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9763525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9763591Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9763821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9763894Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9764121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9764195Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9764419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9764558Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9764561Z 
2025-09-07T09:16:26.9764660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9764843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9764902Z     return mod(**inputs)
2025-09-07T09:16:26.9765134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9765200Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9765432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9765494Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9765711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9765776Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9766019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9766128Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9766361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9766472Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9766698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9766772Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9766777Z 
2025-09-07T09:16:26.9766874Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9767065Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9767129Z     return mod(**inputs)
2025-09-07T09:16:26.9767363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9767433Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9767666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9767736Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9767955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9768028Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9768292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9768366Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9768598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9768670Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9768935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9769044Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9769047Z 
2025-09-07T09:16:26.9769142Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9769334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9769389Z     return mod(**inputs)
2025-09-07T09:16:26.9769621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9769691Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9769930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9770001Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9770218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9770289Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9770517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9770588Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9770818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9770891Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9771122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9771213Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9771216Z 
2025-09-07T09:16:26.9771305Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9771495Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9771552Z     return mod(**inputs)
2025-09-07T09:16:26.9771825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9771888Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9772123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9772222Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9772435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9772506Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9772735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9772811Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9773038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9773109Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9773341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9773436Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9773441Z 
2025-09-07T09:16:26.9773535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9773718Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9773774Z     return mod(**inputs)
2025-09-07T09:16:26.9774052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9774119Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9774389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9774453Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9774674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9774743Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9774970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9775046Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9775271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9775344Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9775572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9775641Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9775645Z 
2025-09-07T09:16:26.9775720Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9775808Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9775999Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9776054Z     return mod(**inputs)
2025-09-07T09:16:26.9776287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9776355Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9776584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9776653Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9776867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9776937Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9777164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9777247Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9777516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9777601Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9777867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9777943Z     return self.weight * hidden_states
2025-09-07T09:16:26.9777947Z 
2025-09-07T09:16:26.9778084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9778283Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9778341Z     return mod(**inputs)
2025-09-07T09:16:26.9778577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9778639Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9778868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9778932Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9779141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9779216Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9779441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9779562Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9779794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9779904Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9780173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9780244Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9780247Z 
2025-09-07T09:16:26.9780341Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9780527Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9780588Z     return mod(**inputs)
2025-09-07T09:16:26.9780824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9780887Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9781123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9781186Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9781397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9781474Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9781700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9781785Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9782012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9782124Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9782352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9782427Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9782430Z 
2025-09-07T09:16:26.9782524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9782711Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9782771Z     return mod(**inputs)
2025-09-07T09:16:26.9783001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9783100Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9783334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9783430Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9783648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9783718Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9783952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9784028Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9784255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9784359Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9784584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9784657Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9784662Z 
2025-09-07T09:16:26.9784733Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9784820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9785009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9785098Z     return mod(**inputs)
2025-09-07T09:16:26.9785337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9785400Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9785659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9785727Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9785941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9786014Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9786244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9786317Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9786546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9786644Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9786870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9786940Z     return self.weight * hidden_states
2025-09-07T09:16:26.9786943Z 
2025-09-07T09:16:26.9787039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9787226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9787282Z     return mod(**inputs)
2025-09-07T09:16:26.9787516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9787579Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9787812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9787876Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9788089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9788159Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9788388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9788464Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9788728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9788806Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9789096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9789167Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9789170Z 
2025-09-07T09:16:26.9789264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9789449Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9789510Z     return mod(**inputs)
2025-09-07T09:16:26.9789740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9789803Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9790036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9790098Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9790312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9790380Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9790605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9790712Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9790940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9791015Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9791274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9791348Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9791352Z 
2025-09-07T09:16:26.9791440Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9791621Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9791685Z     return mod(**inputs)
2025-09-07T09:16:26.9791914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9791980Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9792208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9792268Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9792482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9792549Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9792781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9792851Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9793082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9793155Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9793380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9793503Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9793507Z 
2025-09-07T09:16:26.9793592Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9793777Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9793833Z     return mod(**inputs)
2025-09-07T09:16:26.9794063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9794167Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9794395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9794515Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9794725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9794794Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9795024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9795093Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9795321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9795394Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9795626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9795771Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9795775Z 
2025-09-07T09:16:26.9795866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9796053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9796108Z     return mod(**inputs)
2025-09-07T09:16:26.9796373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9796437Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9796668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9796733Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9796978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9797051Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9797280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9797356Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9797581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9797652Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9797882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9798019Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9798022Z 
2025-09-07T09:16:26.9798115Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9798299Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9798355Z     return mod(**inputs)
2025-09-07T09:16:26.9798588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9798650Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9798882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9798943Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9799156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9799225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9799455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9799527Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9799751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9799859Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9800084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9800187Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9800191Z 
2025-09-07T09:16:26.9800280Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9800465Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9800523Z     return mod(**inputs)
2025-09-07T09:16:26.9800757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9800820Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9801054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9801116Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9801333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9801397Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9801627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9801699Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9801956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9802031Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9802258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9802391Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9802395Z 
2025-09-07T09:16:26.9802484Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9802670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9802730Z     return mod(**inputs)
2025-09-07T09:16:26.9802961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9803031Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9803421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9803486Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9803708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9803780Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9804014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9804087Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9804319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9804392Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9804623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9804730Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9804734Z 
2025-09-07T09:16:26.9804828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9805017Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9805072Z     return mod(**inputs)
2025-09-07T09:16:26.9805307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9805374Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9805682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9805752Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9805970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9806090Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9806329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9806403Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9806648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9806724Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9806961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9807064Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9807067Z 
2025-09-07T09:16:26.9807162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9807359Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9807422Z     return mod(**inputs)
2025-09-07T09:16:26.9807663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9807775Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9808013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9808086Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9808343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9808423Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9808661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9808737Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9808972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9809046Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9809284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9809355Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9809359Z 
2025-09-07T09:16:26.9809461Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9809649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9809712Z     return mod(**inputs)
2025-09-07T09:16:26.9809952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9810018Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9810253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9810318Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9810531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9810604Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9810834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9810911Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9811138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9811267Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9811306Z 
2025-09-07T09:16:26.9811380Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9811471Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9811660Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9811749Z     return mod(**inputs)
2025-09-07T09:16:26.9811987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9812052Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9812284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9812355Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9812569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9812645Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9812877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9812960Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9813198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9813286Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9813548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9813617Z     return self.weight * hidden_states
2025-09-07T09:16:26.9813620Z 
2025-09-07T09:16:26.9813715Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9813899Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9813986Z     return mod(**inputs)
2025-09-07T09:16:26.9814228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9814296Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9814528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9814595Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9814807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9814884Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9815116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9815208Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9815435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9815546Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9815781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9815852Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9815857Z 
2025-09-07T09:16:26.9815956Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9816145Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9816207Z     return mod(**inputs)
2025-09-07T09:16:26.9816439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9816505Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9816740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9816804Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9817021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9817138Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9817367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9817484Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9817711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9817818Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9818114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9818193Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9818196Z 
2025-09-07T09:16:26.9818289Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9818473Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9818536Z     return mod(**inputs)
2025-09-07T09:16:26.9818764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9818835Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9819064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9819126Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9819377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9819448Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9819683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9819789Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9820020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9820128Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9820355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9820431Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9820434Z 
2025-09-07T09:16:26.9820505Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9820600Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9820784Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9820841Z     return mod(**inputs)
2025-09-07T09:16:26.9821078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9821139Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9821376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9821437Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9821654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9821726Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9821954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9822034Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9822266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9822364Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9822598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9822702Z     return self.weight * hidden_states
2025-09-07T09:16:26.9822705Z 
2025-09-07T09:16:26.9822801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9822986Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9823084Z     return mod(**inputs)
2025-09-07T09:16:26.9823316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9823381Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9823617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9823684Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9823903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9823971Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9824200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9824277Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9824503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9824582Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9824804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9824905Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9824914Z 
2025-09-07T09:16:26.9825006Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9825189Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9825284Z     return mod(**inputs)
2025-09-07T09:16:26.9825518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9825588Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9825820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9825885Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9826104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9826170Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9826402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9826473Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9826699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9826773Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9826999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9827075Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9827079Z 
2025-09-07T09:16:26.9827167Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9827354Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9827410Z     return mod(**inputs)
2025-09-07T09:16:26.9827643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9827712Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9827941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9828007Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9828219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9828320Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9828553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9828659Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9828896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9828970Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9829200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9829325Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9829329Z 
2025-09-07T09:16:26.9829422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9829613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9829673Z     return mod(**inputs)
2025-09-07T09:16:26.9829912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9829977Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9830209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9830281Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9830534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9830608Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9830836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9830909Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9831172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9831245Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9831476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9831622Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9831626Z 
2025-09-07T09:16:26.9831721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9831906Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9831964Z     return mod(**inputs)
2025-09-07T09:16:26.9832199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9832259Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9832492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9832558Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9832770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9832844Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9833070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9833146Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9833376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9833450Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9833681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9833818Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9833822Z 
2025-09-07T09:16:26.9833950Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9834134Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9834194Z     return mod(**inputs)
2025-09-07T09:16:26.9834460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9834522Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9834763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9834824Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9835040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9835107Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9835334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9835410Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9835636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9835712Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9835940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9836008Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9836049Z 
2025-09-07T09:16:26.9836140Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9836322Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9836383Z     return mod(**inputs)
2025-09-07T09:16:26.9836660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9836731Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9836962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9837024Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9837240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9837306Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9837539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9837610Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9837835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9837911Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9838136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9838241Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9838245Z 
2025-09-07T09:16:26.9838334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9838523Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9838578Z     return mod(**inputs)
2025-09-07T09:16:26.9838808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9838876Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9839109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9839176Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9839389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9839454Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9839723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9839791Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9840060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9840131Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9840359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9840463Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9840466Z 
2025-09-07T09:16:26.9840554Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9840742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9840799Z     return mod(**inputs)
2025-09-07T09:16:26.9841038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9841099Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9841329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9841397Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9841609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9841711Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9841940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9842011Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9842276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9842348Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9842584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9842675Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9842680Z 
2025-09-07T09:16:26.9842769Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9842956Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9843013Z     return mod(**inputs)
2025-09-07T09:16:26.9843251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9843312Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9843549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9843610Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9843823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9843896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9844125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9844203Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9844426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9844500Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9844731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9844798Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9844801Z 
2025-09-07T09:16:26.9844883Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9844972Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9845188Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9845250Z     return mod(**inputs)
2025-09-07T09:16:26.9845481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9845581Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9845813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9845884Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9846101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9846170Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9846401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9846483Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9846716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9846801Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9847029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9847103Z     return self.weight * hidden_states
2025-09-07T09:16:26.9847106Z 
2025-09-07T09:16:26.9847228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9847427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9847491Z     return mod(**inputs)
2025-09-07T09:16:26.9847722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9847823Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9848057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9848125Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9848338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9848409Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9848633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9848711Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9848942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9849051Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9849283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9849355Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9849358Z 
2025-09-07T09:16:26.9849445Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9849631Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9849691Z     return mod(**inputs)
2025-09-07T09:16:26.9849923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9849984Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9850217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9850277Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9850487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9850560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9850824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9850904Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9851129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9851264Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9851496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9851568Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9851571Z 
2025-09-07T09:16:26.9851661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9851842Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9851900Z     return mod(**inputs)
2025-09-07T09:16:26.9852129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9852191Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9852428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9852492Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9852703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9852802Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9853032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9853111Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9853370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9853472Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9853698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9853767Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9853771Z 
2025-09-07T09:16:26.9853841Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9853928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9854117Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9854174Z     return mod(**inputs)
2025-09-07T09:16:26.9854406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9854474Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9854706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9854777Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9854990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9855061Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9855295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9855365Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9855600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9855698Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9855935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9856001Z     return self.weight * hidden_states
2025-09-07T09:16:26.9856003Z 
2025-09-07T09:16:26.9856093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9856317Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9856375Z     return mod(**inputs)
2025-09-07T09:16:26.9856613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9856708Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9856940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9857018Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9857236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9857309Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9857542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9857617Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9857846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9857914Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9858188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9858259Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9858262Z 
2025-09-07T09:16:26.9858358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9858589Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9858646Z     return mod(**inputs)
2025-09-07T09:16:26.9858884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9858977Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9859217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9859279Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9859494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9859568Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9859796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9859875Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9860100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9860171Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9860398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9860465Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9860470Z 
2025-09-07T09:16:26.9860560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9860740Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9860804Z     return mod(**inputs)
2025-09-07T09:16:26.9861035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9861101Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9861338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9861398Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9861613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9861683Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9861913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9862022Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9862250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9862358Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9862586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9862710Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9862714Z 
2025-09-07T09:16:26.9862803Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9862986Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9863047Z     return mod(**inputs)
2025-09-07T09:16:26.9863279Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9863347Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9863577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9863639Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9863854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9863920Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9864183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9864252Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9864488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9864591Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9864820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9864967Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9864970Z 
2025-09-07T09:16:26.9865061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9865246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9865301Z     return mod(**inputs)
2025-09-07T09:16:26.9865532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9865598Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9865825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9865890Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9866104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9866176Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9866400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9866471Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9866698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9866766Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9866995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9867133Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9867136Z 
2025-09-07T09:16:26.9867222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9867409Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9867504Z     return mod(**inputs)
2025-09-07T09:16:26.9867740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9867834Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9868063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9868129Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9868345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9868415Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9868640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9868716Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9868946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9869019Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9869248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9869315Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9869320Z 
2025-09-07T09:16:26.9869418Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9869632Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9869690Z     return mod(**inputs)
2025-09-07T09:16:26.9869924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9869991Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9870254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9870317Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9870529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9870601Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9870831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9870906Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9871134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9871204Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9871431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9871530Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9871533Z 
2025-09-07T09:16:26.9871632Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9871817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9871880Z     return mod(**inputs)
2025-09-07T09:16:26.9872112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9872174Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9872408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9872469Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9872684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9879714Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9880032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9880183Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9880448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9880536Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9880824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9880931Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9880936Z 
2025-09-07T09:16:26.9881045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9881253Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9881323Z     return mod(**inputs)
2025-09-07T09:16:26.9881577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9881650Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9881888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9881953Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9882187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9882262Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9882534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9882616Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9882860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9882938Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9883202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9883311Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9883315Z 
2025-09-07T09:16:26.9883413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9883609Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9883678Z     return mod(**inputs)
2025-09-07T09:16:26.9883916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9883988Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9884216Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9884281Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9884505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9884583Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9884824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9884896Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9885125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9885197Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9885423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9885495Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9885498Z 
2025-09-07T09:16:26.9885588Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9885775Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9885832Z     return mod(**inputs)
2025-09-07T09:16:26.9886065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9886187Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9886417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9886528Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9886742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9886811Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9887041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9887110Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9887340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9887464Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9887468Z 
2025-09-07T09:16:26.9887543Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9887631Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9887815Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9887875Z     return mod(**inputs)
2025-09-07T09:16:26.9888103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9888202Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9888437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9888501Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9888750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9888819Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9889053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9889135Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9889365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9889457Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9889682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9889753Z     return self.weight * hidden_states
2025-09-07T09:16:26.9889756Z 
2025-09-07T09:16:26.9889848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9890040Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9890097Z     return mod(**inputs)
2025-09-07T09:16:26.9890328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9890394Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9890624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9890689Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9890899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9890967Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9891196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9891274Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9891503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9891612Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9891873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9891945Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9891979Z 
2025-09-07T09:16:26.9892068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9892253Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9892309Z     return mod(**inputs)
2025-09-07T09:16:26.9892541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9892603Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9892832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9892897Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9893108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9893179Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9893404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9893484Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9893710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9893844Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9894073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9894144Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9894147Z 
2025-09-07T09:16:26.9894268Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9894455Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9894512Z     return mod(**inputs)
2025-09-07T09:16:26.9894745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9894811Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9895041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9895101Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9895320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9895396Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9895625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9895707Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9895939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9896041Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9896277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9896351Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9896354Z 
2025-09-07T09:16:26.9896434Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9896524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9896714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9896772Z     return mod(**inputs)
2025-09-07T09:16:26.9897003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9897071Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9897341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9897412Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9897624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9897724Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9897958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9898116Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9898348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9898444Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9898699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9898775Z     return self.weight * hidden_states
2025-09-07T09:16:26.9898778Z 
2025-09-07T09:16:26.9898872Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9899068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9899131Z     return mod(**inputs)
2025-09-07T09:16:26.9899372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9899445Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9899711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9899783Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9899998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9900100Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9900340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9900415Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9900649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9900727Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9900954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9901031Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9901034Z 
2025-09-07T09:16:26.9901125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9901316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9901374Z     return mod(**inputs)
2025-09-07T09:16:26.9901609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9901675Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9901906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9901976Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9902189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9902260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9902489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9902561Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9902793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9902864Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9903314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9903384Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9903387Z 
2025-09-07T09:16:26.9903543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9903730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9903788Z     return mod(**inputs)
2025-09-07T09:16:26.9904026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9904088Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9904321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9904383Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9904596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9904670Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9904898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9904975Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9905200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9905269Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9905553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9905677Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9905681Z 
2025-09-07T09:16:26.9905775Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9906000Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9906065Z     return mod(**inputs)
2025-09-07T09:16:26.9906303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9906366Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9906601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9906662Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9906879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9906945Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9907169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9907243Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9907471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9907547Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9907774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9907923Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9907931Z 
2025-09-07T09:16:26.9908019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9908203Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9908267Z     return mod(**inputs)
2025-09-07T09:16:26.9908496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9908564Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9908794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9908916Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9909132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9909232Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9909460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9909529Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9909760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9909833Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9910060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9910201Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9910206Z 
2025-09-07T09:16:26.9910295Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9910480Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9910536Z     return mod(**inputs)
2025-09-07T09:16:26.9910767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9910834Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9911094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9911161Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9911374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9911440Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9911704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9911775Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9912007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9912077Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9912302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9912374Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9912378Z 
2025-09-07T09:16:26.9912465Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9912654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9912710Z     return mod(**inputs)
2025-09-07T09:16:26.9912960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9913026Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9913270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9913336Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9913552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9913624Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9913854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9913925Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9914158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9914229Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9914460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9914597Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9914601Z 
2025-09-07T09:16:26.9914697Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9914925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9914984Z     return mod(**inputs)
2025-09-07T09:16:26.9915220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9915285Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9915517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9915580Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9915793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9915866Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9916098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9916174Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9916400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9916474Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9916736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9916832Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9916835Z 
2025-09-07T09:16:26.9916931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9917152Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9917214Z     return mod(**inputs)
2025-09-07T09:16:26.9917450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9917512Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9917746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9917810Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9918035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9918105Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9918343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9918417Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9918644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9918720Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9918949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9919042Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9919051Z 
2025-09-07T09:16:26.9919141Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9919325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9919390Z     return mod(**inputs)
2025-09-07T09:16:26.9919620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9919689Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9919920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9919985Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9920245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9920315Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9920547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9920656Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9920888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9920960Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9921189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9921256Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9921259Z 
2025-09-07T09:16:26.9921334Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9921427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9921615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9921679Z     return mod(**inputs)
2025-09-07T09:16:26.9921913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9921979Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9922206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9922303Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9922525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9922591Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9922857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9922939Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9923168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9923258Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9923485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9923555Z     return self.weight * hidden_states
2025-09-07T09:16:26.9923559Z 
2025-09-07T09:16:26.9923651Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9923836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9923894Z     return mod(**inputs)
2025-09-07T09:16:26.9924126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9924191Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9924418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9924483Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9924696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9924764Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9924996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9925072Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9925301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9925409Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9925636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9925739Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9925742Z 
2025-09-07T09:16:26.9925831Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9926018Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9926109Z     return mod(**inputs)
2025-09-07T09:16:26.9926339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9926403Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9926634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9926701Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9926911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9926978Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9927205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9927282Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9927513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9927616Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9927878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9927951Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9927954Z 
2025-09-07T09:16:26.9928042Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9928226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9928328Z     return mod(**inputs)
2025-09-07T09:16:26.9928565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9928628Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9928863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9928927Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9929139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9929207Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9929435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9929513Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9929741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9929841Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9930074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9930143Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9930148Z 
2025-09-07T09:16:26.9930222Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9930310Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9930491Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9930549Z     return mod(**inputs)
2025-09-07T09:16:26.9930779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9930845Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9931074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9931136Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9931405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9931477Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9931752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9931826Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9932064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9932163Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9932391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9932459Z     return self.weight * hidden_states
2025-09-07T09:16:26.9932464Z 
2025-09-07T09:16:26.9932552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9932742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9932799Z     return mod(**inputs)
2025-09-07T09:16:26.9933029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9933096Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9933325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9933422Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9933639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9933706Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9933963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9934037Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9934271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9934340Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9934569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9934637Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9934641Z 
2025-09-07T09:16:26.9934729Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9934914Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9934969Z     return mod(**inputs)
2025-09-07T09:16:26.9935203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9935267Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9935496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9935559Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9935771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9935843Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9936074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9936146Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9936372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9936441Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9936669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9936736Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9936773Z 
2025-09-07T09:16:26.9936866Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9937048Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9937138Z     return mod(**inputs)
2025-09-07T09:16:26.9937378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9937443Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9937685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9937749Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9938045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9938124Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9938362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9938438Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9938664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9938739Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9938966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9939125Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9939129Z 
2025-09-07T09:16:26.9939228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9939420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9939482Z     return mod(**inputs)
2025-09-07T09:16:26.9939747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9939813Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9940046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9940111Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9940328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9940395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9940625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9940696Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9940925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9941002Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9941230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9941384Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9941387Z 
2025-09-07T09:16:26.9941479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9941664Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9941722Z     return mod(**inputs)
2025-09-07T09:16:26.9941951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9942019Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9942247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9942308Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9942524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9942625Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9942858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9942959Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9943191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9943259Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9943490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9943627Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9943630Z 
2025-09-07T09:16:26.9943718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9943907Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9943965Z     return mod(**inputs)
2025-09-07T09:16:26.9944193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9944259Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9944488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9944553Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9944795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9944864Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9945093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9945402Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9945638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9945710Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9945943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9946015Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9946019Z 
2025-09-07T09:16:26.9946106Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9946293Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9946350Z     return mod(**inputs)
2025-09-07T09:16:26.9946582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9946643Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9946871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9946939Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9947148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9947220Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9947447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9947517Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9947743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9947816Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9948043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9948144Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9948147Z 
2025-09-07T09:16:26.9948242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9948464Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9948519Z     return mod(**inputs)
2025-09-07T09:16:26.9948784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9948850Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9949086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9949147Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9949362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9949434Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9949663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9949736Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9949963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9950038Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9950266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9950357Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9950360Z 
2025-09-07T09:16:26.9950498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9950683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9950743Z     return mod(**inputs)
2025-09-07T09:16:26.9951005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9951069Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9951309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9951372Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9951587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9951653Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9951889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9951959Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9952187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9952262Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9952492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9952593Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9952596Z 
2025-09-07T09:16:26.9952687Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9952872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9952933Z     return mod(**inputs)
2025-09-07T09:16:26.9953161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9953229Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9953458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9953519Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9953733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9953799Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9954070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9954141Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9954403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9954473Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9954703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9954774Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9954778Z 
2025-09-07T09:16:26.9954867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9955055Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9955113Z     return mod(**inputs)
2025-09-07T09:16:26.9955342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9955408Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9955635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9955704Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9955918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9956017Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9956251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9956322Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9956585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:26.9956710Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:26.9956715Z 
2025-09-07T09:16:26.9956787Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9956875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9957060Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9957122Z     return mod(**inputs)
2025-09-07T09:16:26.9957351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9957416Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9957643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9957704Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9957918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9957983Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9958213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9958294Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9958519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9958606Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9958833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9958902Z     return self.weight * hidden_states
2025-09-07T09:16:26.9958905Z 
2025-09-07T09:16:26.9958992Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9959177Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9959234Z     return mod(**inputs)
2025-09-07T09:16:26.9959463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9959561Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9959790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9959888Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9960101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9960168Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9960394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9960472Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9960703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9960811Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9961042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9961112Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9961117Z 
2025-09-07T09:16:26.9961206Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9961394Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9961449Z     return mod(**inputs)
2025-09-07T09:16:26.9961713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9961777Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9962008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9962107Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9962322Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9962395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9962625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9962702Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9962932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9963034Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9963267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9963338Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9963341Z 
2025-09-07T09:16:26.9963432Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9963615Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9963674Z     return mod(**inputs)
2025-09-07T09:16:26.9963907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9963970Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9964199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9964260Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9964471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9964541Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9964767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9964848Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9965111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9965219Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9965446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9965549Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9965552Z 
2025-09-07T09:16:26.9965627Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9965717Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9965900Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9965958Z     return mod(**inputs)
2025-09-07T09:16:26.9966187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9966253Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9966483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9966546Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9966757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9966824Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9967053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9967156Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9967390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:26.9967486Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9967748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9967819Z     return self.weight * hidden_states
2025-09-07T09:16:26.9967824Z 
2025-09-07T09:16:26.9967912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9968115Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9968180Z     return mod(**inputs)
2025-09-07T09:16:26.9968415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9968477Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9968705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9968767Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9968978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9969046Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9969272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9969340Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9969566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9969638Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9969866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:26.9969934Z     query_states = self.q(hidden_states)
2025-09-07T09:16:26.9969937Z 
2025-09-07T09:16:26.9970026Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9970207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9970264Z     return mod(**inputs)
2025-09-07T09:16:26.9970495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9970592Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9970822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9970929Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9971140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9971208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9971435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9971505Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9971728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9971798Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9972023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:26.9972092Z     key_states = self.k(current_states)
2025-09-07T09:16:26.9972095Z 
2025-09-07T09:16:26.9972186Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9972365Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9972421Z     return mod(**inputs)
2025-09-07T09:16:26.9972687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9972749Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9972979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9973039Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9973305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9973372Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9973598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9973674Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9973899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9973973Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9974199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:26.9974319Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:26.9974327Z 
2025-09-07T09:16:26.9974417Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9974601Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9974667Z     return mod(**inputs)
2025-09-07T09:16:26.9974897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9974966Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9975198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9975260Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9975478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9975545Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9975776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9975847Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9976074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9976186Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9976414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9976602Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9976605Z 
2025-09-07T09:16:26.9976694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9976880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9976937Z     return mod(**inputs)
2025-09-07T09:16:26.9977167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9977236Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9977466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9977533Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9977744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9977812Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9978106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9978176Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9978438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9978508Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9978736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:26.9978907Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:26.9978912Z 
2025-09-07T09:16:26.9979003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9979192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9979248Z     return mod(**inputs)
2025-09-07T09:16:26.9979486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9979549Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9979779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9979845Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9980056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9980129Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9980357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9980429Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9980660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9980730Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9980962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:26.9981033Z     value_states = self.v(current_states)
2025-09-07T09:16:26.9981038Z 
2025-09-07T09:16:26.9981131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9981312Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9981370Z     return mod(**inputs)
2025-09-07T09:16:26.9981602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9981666Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9981939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9982003Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9982251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9982322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9982552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9982628Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9982854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9982926Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9983154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9983254Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9983258Z 
2025-09-07T09:16:26.9983353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9983537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9983597Z     return mod(**inputs)
2025-09-07T09:16:26.9983828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9983924Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9984161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9984224Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9984470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9984538Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9984768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9984843Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9985071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9985146Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9985374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:26.9985467Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:26.9985474Z 
2025-09-07T09:16:26.9985566Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9985752Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9985815Z     return mod(**inputs)
2025-09-07T09:16:26.9986045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9986108Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9986337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9986403Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9986618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9986684Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9986920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9986990Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9987218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9987325Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9987552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:26.9987649Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:26.9987686Z 
2025-09-07T09:16:26.9987773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9987963Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9988021Z     return mod(**inputs)
2025-09-07T09:16:26.9988256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9988325Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9988565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9988632Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9988846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9988912Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9989143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:26.9989220Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:26.9989471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:26.9989578Z     attention_output = self.SelfAttention(
2025-09-07T09:16:26.9989811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:26.9989883Z     attn_output = self.o(attn_output)
2025-09-07T09:16:26.9989887Z 
2025-09-07T09:16:26.9989990Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9990093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9990282Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9990343Z     return mod(**inputs)
2025-09-07T09:16:26.9990573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9990637Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9990872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9990934Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9991147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9991213Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9991441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9991525Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9991752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:26.9991842Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:26.9992068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:26.9992134Z     return self.weight * hidden_states
2025-09-07T09:16:26.9992142Z 
2025-09-07T09:16:26.9992232Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9992416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9992475Z     return mod(**inputs)
2025-09-07T09:16:26.9992704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9992771Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9993052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9993114Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9993331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9993433Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9993666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9993748Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9993975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9994087Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9994316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:26.9994390Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:26.9994394Z 
2025-09-07T09:16:26.9994482Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9994668Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9994727Z     return mod(**inputs)
2025-09-07T09:16:26.9994959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9995055Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9995291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9995357Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9995600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9995668Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9995903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9995977Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9996211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9996312Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9996545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:26.9996617Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:26.9996621Z 
2025-09-07T09:16:26.9996710Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9996896Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9996952Z     return mod(**inputs)
2025-09-07T09:16:26.9997184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9997248Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9997476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9997544Z     layer_outputs = layer_module(
2025-09-07T09:16:26.9997754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:26.9997828Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:26.9998052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:26.9998126Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:26.9998359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:26.9998458Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:26.9998727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:26.9998795Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:26.9998830Z 
2025-09-07T09:16:26.9998901Z cudagraph partition due to non gpu ops
2025-09-07T09:16:26.9998994Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:26.9999177Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:26.9999238Z     return mod(**inputs)
2025-09-07T09:16:26.9999468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:26.9999536Z     encoder_outputs = self.encoder(
2025-09-07T09:16:26.9999766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:26.9999828Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0000046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0000112Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0000345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0000414Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0000676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0000778Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0001006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0001077Z     return self.weight * hidden_states
2025-09-07T09:16:27.0001112Z 
2025-09-07T09:16:27.0001205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0001390Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0001451Z     return mod(**inputs)
2025-09-07T09:16:27.0001681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0001750Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0001979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0002048Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0002260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0002325Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0002561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0002631Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0002861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0002930Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0003294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0003367Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0003370Z 
2025-09-07T09:16:27.0003460Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0003649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0003705Z     return mod(**inputs)
2025-09-07T09:16:27.0003935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0004002Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0004232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0004352Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0004565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0004681Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0004909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0004983Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0005215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0005286Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0005520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0005589Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0005593Z 
2025-09-07T09:16:27.0005683Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0005871Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0005929Z     return mod(**inputs)
2025-09-07T09:16:27.0006162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0006225Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0006503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0006570Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0006783Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0006896Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0007125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0007202Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0007432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0007505Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0007736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0007854Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0007858Z 
2025-09-07T09:16:27.0007952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0008140Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0008201Z     return mod(**inputs)
2025-09-07T09:16:27.0008441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0008503Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0008737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0008800Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0009014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0009080Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0009307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0009381Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0009605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0009679Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0009906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0010086Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0010089Z 
2025-09-07T09:16:27.0010212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0010393Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0010454Z     return mod(**inputs)
2025-09-07T09:16:27.0010684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0010747Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0010980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0011042Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0011259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0011327Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0011558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0011631Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0011857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0011931Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0012189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0012328Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0012331Z 
2025-09-07T09:16:27.0012418Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0012646Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0012709Z     return mod(**inputs)
2025-09-07T09:16:27.0012938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0013006Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0013233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0013295Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0013514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0013577Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0013805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0013876Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0014108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0014177Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0014406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0014483Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0014486Z 
2025-09-07T09:16:27.0014574Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0014762Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0014818Z     return mod(**inputs)
2025-09-07T09:16:27.0015052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0015117Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0015349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0015452Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0015666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0015736Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0015998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0016068Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0016301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0016372Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0016602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0016703Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0016706Z 
2025-09-07T09:16:27.0016796Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0016986Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0017044Z     return mod(**inputs)
2025-09-07T09:16:27.0017280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0017345Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0017607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0017674Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0017886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0017958Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0018266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0018341Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0018567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0018639Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0018870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0018960Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0018964Z 
2025-09-07T09:16:27.0019061Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0019242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0019299Z     return mod(**inputs)
2025-09-07T09:16:27.0019534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0019596Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0019831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0019892Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0020107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0020173Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0020398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0020474Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0020702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0020776Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0021006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0021132Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0021135Z 
2025-09-07T09:16:27.0021229Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0021410Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0021505Z     return mod(**inputs)
2025-09-07T09:16:27.0021739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0021803Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0022035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0022097Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0022314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0022391Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0022625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0022694Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0022921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0022996Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0023255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0023327Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0023331Z 
2025-09-07T09:16:27.0023419Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0023601Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0023691Z     return mod(**inputs)
2025-09-07T09:16:27.0023924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0023995Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0024225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0024289Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0024506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0024572Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0024802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0024869Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0025102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.0025227Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0025232Z 
2025-09-07T09:16:27.0025303Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0025398Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0025578Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0025639Z     return mod(**inputs)
2025-09-07T09:16:27.0025868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0025935Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0026168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0026230Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0026445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0026511Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0026775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0026866Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0027129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0027223Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0027449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0027521Z     return self.weight * hidden_states
2025-09-07T09:16:27.0027525Z 
2025-09-07T09:16:27.0027617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0027802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0027863Z     return mod(**inputs)
2025-09-07T09:16:27.0028093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0028163Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0028394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0028457Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0028673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0028773Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0029009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0029088Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0029353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0029461Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0029688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0029765Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0029770Z 
2025-09-07T09:16:27.0029859Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0030047Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0030102Z     return mod(**inputs)
2025-09-07T09:16:27.0030332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0030401Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0030632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0030701Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0030915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0030986Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0031214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0031295Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0031531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0031634Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0031865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0031936Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0031939Z 
2025-09-07T09:16:27.0032026Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0032215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0032308Z     return mod(**inputs)
2025-09-07T09:16:27.0032547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0032644Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0032874Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0032940Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0033155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0033226Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0033454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0033535Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0033765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0033867Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0034101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0034172Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0034175Z 
2025-09-07T09:16:27.0034249Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0034382Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0034569Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0034634Z     return mod(**inputs)
2025-09-07T09:16:27.0034895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0034964Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0035196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0035259Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0035478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0035545Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0035770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0035842Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0036075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0036172Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0036400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0036472Z     return self.weight * hidden_states
2025-09-07T09:16:27.0036475Z 
2025-09-07T09:16:27.0036563Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0036750Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0036808Z     return mod(**inputs)
2025-09-07T09:16:27.0037038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0037106Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0037335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0037399Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0037614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0037679Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0037945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0038016Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0038246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0038351Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0038586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0038656Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0038660Z 
2025-09-07T09:16:27.0038749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0038942Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0038998Z     return mod(**inputs)
2025-09-07T09:16:27.0039236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0039301Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0039530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0039598Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0039818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0039889Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0040146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0040219Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0040481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0040554Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0040786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0040852Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0040856Z 
2025-09-07T09:16:27.0040951Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0041133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0041191Z     return mod(**inputs)
2025-09-07T09:16:27.0041429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0041493Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0041732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0041794Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0042009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0042080Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0042305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0042383Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0042608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0042681Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0042911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0043028Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0043033Z 
2025-09-07T09:16:27.0043128Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0043313Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0043410Z     return mod(**inputs)
2025-09-07T09:16:27.0043643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0043707Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0043976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0044041Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0044259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0044327Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0044557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0044633Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0044857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0044936Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0045165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0045313Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0045316Z 
2025-09-07T09:16:27.0045403Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0045619Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0045681Z     return mod(**inputs)
2025-09-07T09:16:27.0045914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0045984Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0046248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0046313Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0046531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0046600Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0046828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0046896Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0047122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0047201Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0047425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0047565Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0047570Z 
2025-09-07T09:16:27.0047659Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0047849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0047907Z     return mod(**inputs)
2025-09-07T09:16:27.0048139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0048207Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0048439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0048504Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0048716Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0048781Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0049014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0049118Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0049352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0049458Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0049689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0049758Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0049762Z 
2025-09-07T09:16:27.0049850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0050043Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0050099Z     return mod(**inputs)
2025-09-07T09:16:27.0050334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0050398Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0050626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0050694Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0050906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0050973Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0051237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0051305Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0051538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0051607Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0051866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0051965Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0051969Z 
2025-09-07T09:16:27.0052062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0052246Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0052305Z     return mod(**inputs)
2025-09-07T09:16:27.0052538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0052601Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0052833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0052894Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0053106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0053175Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0053404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0053476Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0053700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0053771Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0054004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0054093Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0054096Z 
2025-09-07T09:16:27.0054188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0054374Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0054433Z     return mod(**inputs)
2025-09-07T09:16:27.0054698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0054762Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0054997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0055128Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0055346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0055413Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0055642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0055716Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0055944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0056024Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0056252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0056351Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0056356Z 
2025-09-07T09:16:27.0056449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0056630Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0056727Z     return mod(**inputs)
2025-09-07T09:16:27.0056962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0057035Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0057298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0057362Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0057582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0057649Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0057892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0058002Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0058231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0058307Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0058536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0058608Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0058612Z 
2025-09-07T09:16:27.0058685Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0058783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0058970Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0059029Z     return mod(**inputs)
2025-09-07T09:16:27.0059265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0059329Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0059566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0059631Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0059844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0059918Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0060147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0060235Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0060503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0060592Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0060856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0060927Z     return self.weight * hidden_states
2025-09-07T09:16:27.0060930Z 
2025-09-07T09:16:27.0061026Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0061213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0061274Z     return mod(**inputs)
2025-09-07T09:16:27.0061512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0061575Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0061813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0061876Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0062088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0062155Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0062381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0062522Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0062753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0062861Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0063119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0063192Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0063201Z 
2025-09-07T09:16:27.0063288Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0063471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0063536Z     return mod(**inputs)
2025-09-07T09:16:27.0063766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0063834Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0064063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0064124Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0064343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0064410Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0064646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0064729Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0064953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0065061Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0065287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0065361Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0065364Z 
2025-09-07T09:16:27.0065451Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0065644Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0065704Z     return mod(**inputs)
2025-09-07T09:16:27.0065938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0066042Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0066273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0066371Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0066582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0066651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0066884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0066957Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0067191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0067293Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0067521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0067599Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0067604Z 
2025-09-07T09:16:27.0067676Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0067773Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0067957Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0068055Z     return mod(**inputs)
2025-09-07T09:16:27.0068289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0068355Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0068621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0068682Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0068901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0068968Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0069199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0069274Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0069502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0069605Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0069832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0069900Z     return self.weight * hidden_states
2025-09-07T09:16:27.0069912Z 
2025-09-07T09:16:27.0070005Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0070189Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0070252Z     return mod(**inputs)
2025-09-07T09:16:27.0070483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0070553Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0070782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0070848Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0071067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0071133Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0071365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0071436Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0071703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0071780Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0072042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0072116Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0072119Z 
2025-09-07T09:16:27.0072209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0072400Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0072458Z     return mod(**inputs)
2025-09-07T09:16:27.0072686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0072756Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0072987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0073055Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0073266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0073334Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0073568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0073673Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0073909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0073979Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0074238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0074313Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0074318Z 
2025-09-07T09:16:27.0074406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0074598Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0074657Z     return mod(**inputs)
2025-09-07T09:16:27.0074888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0074949Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0075180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0075246Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0075457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0075524Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0075748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0075817Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0076046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0076118Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0076366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0076498Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0076501Z 
2025-09-07T09:16:27.0076595Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0076784Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0076842Z     return mod(**inputs)
2025-09-07T09:16:27.0077074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0077188Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0077428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0077526Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0077740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0077812Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0078041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0078118Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0078343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0078413Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0078640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0078784Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0078788Z 
2025-09-07T09:16:27.0078878Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0079062Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0079118Z     return mod(**inputs)
2025-09-07T09:16:27.0079382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0079444Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0079676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0079736Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0079981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0080059Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0080286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0080364Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0080591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0080664Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0080891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0081030Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0081036Z 
2025-09-07T09:16:27.0081126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0081308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0081369Z     return mod(**inputs)
2025-09-07T09:16:27.0081597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0081665Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0081894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0081955Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0082172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0082237Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0082471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0082539Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0082764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0082873Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0083097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0083204Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0083207Z 
2025-09-07T09:16:27.0083296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0083479Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0083541Z     return mod(**inputs)
2025-09-07T09:16:27.0083769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0083835Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0084065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0084130Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0084340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0084408Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0084639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0084706Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0084967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0085036Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0085262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0085392Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0085396Z 
2025-09-07T09:16:27.0085483Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0085670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0085725Z     return mod(**inputs)
2025-09-07T09:16:27.0085957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0086017Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0086244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0086306Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0086515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0086580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0086805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0086875Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0087101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0087170Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0087396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0087487Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0087491Z 
2025-09-07T09:16:27.0087580Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0087763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0087817Z     return mod(**inputs)
2025-09-07T09:16:27.0088049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0088110Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0088380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0088443Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0088690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0088759Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0088985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0089056Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0089280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0089349Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0089580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0089674Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0089677Z 
2025-09-07T09:16:27.0089767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0089947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0090004Z     return mod(**inputs)
2025-09-07T09:16:27.0090235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0090329Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0090563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0090624Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0090870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0090939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0091169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0091243Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0091469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0091544Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0091771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0091840Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0091843Z 
2025-09-07T09:16:27.0091936Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0092119Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0092183Z     return mod(**inputs)
2025-09-07T09:16:27.0092412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0092476Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0092710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0099830Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0100115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0100208Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0100473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0100554Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0100816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.0100948Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0101023Z 
2025-09-07T09:16:27.0101102Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0101205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0101415Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0101513Z     return mod(**inputs)
2025-09-07T09:16:27.0101773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0101846Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0102083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0102152Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0102377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0102454Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0102691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0102776Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0103009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0103281Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0103597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0103669Z     return self.weight * hidden_states
2025-09-07T09:16:27.0103673Z 
2025-09-07T09:16:27.0103777Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0103976Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0104099Z     return mod(**inputs)
2025-09-07T09:16:27.0104340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0104412Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0104663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0104730Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0104957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0105037Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0105278Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0105367Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0105600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0105713Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0105943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0106021Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0106027Z 
2025-09-07T09:16:27.0106130Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0106331Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0106392Z     return mod(**inputs)
2025-09-07T09:16:27.0106628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0106694Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0106935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0107000Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0107224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0107344Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0107573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0107705Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0107932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0108047Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0108275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0108353Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0108357Z 
2025-09-07T09:16:27.0108451Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0108640Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0108706Z     return mod(**inputs)
2025-09-07T09:16:27.0108945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0109019Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0109264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0109327Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0109588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0109660Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0109900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0110011Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0110248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0110353Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0110584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0110660Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0110663Z 
2025-09-07T09:16:27.0110738Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0110838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0111032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0111095Z     return mod(**inputs)
2025-09-07T09:16:27.0111332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward
2025-09-07T09:16:27.0111398Z     encoder_outputs = self.encoder(
2025-09-07T09:16:27.0111639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward
2025-09-07T09:16:27.0111735Z     hidden_states = self.final_layer_norm(hidden_states)
2025-09-07T09:16:27.0111967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0112040Z     return self.weight * hidden_states
2025-09-07T09:16:27.0112043Z 
2025-09-07T09:16:27.0112134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0112325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0112382Z     return mod(**inputs)
2025-09-07T09:16:27.0112615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0112678Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0112909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0113012Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0113228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0113332Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0113560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0113634Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0113868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0113941Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0114178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0114248Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0114253Z 
2025-09-07T09:16:27.0114351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0114535Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0114594Z     return mod(**inputs)
2025-09-07T09:16:27.0114831Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0114892Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0115163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0115225Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0115443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0115554Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0115785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0115863Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0116090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0116163Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0116394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0116519Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0116523Z 
2025-09-07T09:16:27.0116617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0116801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0116861Z     return mod(**inputs)
2025-09-07T09:16:27.0117094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0117160Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0117393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0117454Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0117670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0117736Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0117964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0118041Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0118266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0118347Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0118576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0118758Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0118766Z 
2025-09-07T09:16:27.0118887Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0119073Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0119134Z     return mod(**inputs)
2025-09-07T09:16:27.0119368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0119431Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0119669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0119732Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0119950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0120018Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0120245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0120320Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0120545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0120620Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0120875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0120946Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0120952Z 
2025-09-07T09:16:27.0121042Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0121255Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0121317Z     return mod(**inputs)
2025-09-07T09:16:27.0121548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0121615Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0121849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0121911Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0122128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0122194Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0122423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0122493Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0122721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0122800Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0123026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0123133Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0123136Z 
2025-09-07T09:16:27.0123226Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0123416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0123473Z     return mod(**inputs)
2025-09-07T09:16:27.0123704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0123773Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0124005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0124103Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0124316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0124381Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0124642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0124711Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0124943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0125014Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0125248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0125349Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0125353Z 
2025-09-07T09:16:27.0125439Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0125625Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0125680Z     return mod(**inputs)
2025-09-07T09:16:27.0125910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0125974Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0126275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0126340Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0126553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0126619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0126888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0126960Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0127196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0127266Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0127493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0127584Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0127588Z 
2025-09-07T09:16:27.0127678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0127860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0127917Z     return mod(**inputs)
2025-09-07T09:16:27.0128151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0128212Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0128445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0128505Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0128715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0128784Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0129010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0129080Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0129305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0129375Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0129604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0129714Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0129717Z 
2025-09-07T09:16:27.0129789Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0129876Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0130092Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0130149Z     return mod(**inputs)
2025-09-07T09:16:27.0130378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0130443Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0130671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0130736Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0130948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0131015Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0131246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0131329Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0131560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0131648Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0131907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0131977Z     return self.weight * hidden_states
2025-09-07T09:16:27.0131980Z 
2025-09-07T09:16:27.0132066Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0132282Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0132338Z     return mod(**inputs)
2025-09-07T09:16:27.0132574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0132636Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0132867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0132931Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0133142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0133210Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0133435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0133512Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0133740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0133846Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0134074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0134146Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0134149Z 
2025-09-07T09:16:27.0134236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0134418Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0134474Z     return mod(**inputs)
2025-09-07T09:16:27.0134705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0134766Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0134996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0135056Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0135303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0135371Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0135597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0135706Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0135940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0136042Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0136269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0136340Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0136345Z 
2025-09-07T09:16:27.0136435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0136617Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0136672Z     return mod(**inputs)
2025-09-07T09:16:27.0136902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0136964Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0137193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0137285Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0137500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0137565Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0137821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0137899Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0138161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0138262Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0138490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0138557Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0138560Z 
2025-09-07T09:16:27.0138652Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0138834Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0138890Z     return mod(**inputs)
2025-09-07T09:16:27.0139117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0139182Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0139409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0139472Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0139684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0139750Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0139977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0140048Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0140271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0140368Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0140593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0140709Z     return self.weight * hidden_states
2025-09-07T09:16:27.0140712Z 
2025-09-07T09:16:27.0140801Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0140988Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0141076Z     return mod(**inputs)
2025-09-07T09:16:27.0141310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0141371Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0141602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0141666Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0141875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0141944Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0142168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0142240Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0142464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0142536Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0142765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0142867Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0142871Z 
2025-09-07T09:16:27.0142964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0143147Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0143201Z     return mod(**inputs)
2025-09-07T09:16:27.0143466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0143531Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0143765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0143826Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0144038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0144103Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0144330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0144402Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0144629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0144700Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0144922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0144989Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0144992Z 
2025-09-07T09:16:27.0145081Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0145263Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0145322Z     return mod(**inputs)
2025-09-07T09:16:27.0145558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0145629Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0145873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0145935Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0146149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0146253Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0146482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0146599Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0146825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0146897Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0147123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0147248Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0147252Z 
2025-09-07T09:16:27.0147340Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0147524Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0147583Z     return mod(**inputs)
2025-09-07T09:16:27.0147814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0147876Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0148107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0148168Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0148417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0148483Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0148712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0148778Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0149041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0149111Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0149337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0149485Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0149489Z 
2025-09-07T09:16:27.0149576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0149766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0149821Z     return mod(**inputs)
2025-09-07T09:16:27.0150049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0150114Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0150344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0150410Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0150619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0150689Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0150917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0150986Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0151215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0151285Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0151513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0151581Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0151584Z 
2025-09-07T09:16:27.0151674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0151898Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0151954Z     return mod(**inputs)
2025-09-07T09:16:27.0152187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0152281Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0152513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0152578Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0152789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0152857Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0153085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0153161Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0153389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0153457Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0153687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0153782Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0153785Z 
2025-09-07T09:16:27.0153907Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0154093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0154149Z     return mod(**inputs)
2025-09-07T09:16:27.0154417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0154480Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0154715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0154777Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0154989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0155059Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0155285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0155359Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0155585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0155657Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0155883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0155975Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0155979Z 
2025-09-07T09:16:27.0156070Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0156251Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0156310Z     return mod(**inputs)
2025-09-07T09:16:27.0156541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0156604Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0156837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0156899Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0157115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0157180Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0157449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0157518Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0157742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0157885Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0158115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0158208Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0158210Z 
2025-09-07T09:16:27.0158296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0158478Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0158539Z     return mod(**inputs)
2025-09-07T09:16:27.0158773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0158839Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0159070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0159134Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0159346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0159442Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0159676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0159745Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0160005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0160078Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0160306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0160375Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0160380Z 
2025-09-07T09:16:27.0160450Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0160545Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0160729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0160786Z     return mod(**inputs)
2025-09-07T09:16:27.0161019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0161082Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0161317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0161380Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0161592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0161659Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0161887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0161962Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0162187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0162286Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0162512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0162579Z     return self.weight * hidden_states
2025-09-07T09:16:27.0162582Z 
2025-09-07T09:16:27.0162673Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0162856Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0162952Z     return mod(**inputs)
2025-09-07T09:16:27.0163180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0163278Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0163525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0163586Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0163804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0163871Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0164100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0164172Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0164398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0164477Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0164702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0164775Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0164779Z 
2025-09-07T09:16:27.0164867Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0165082Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0165144Z     return mod(**inputs)
2025-09-07T09:16:27.0165377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0165477Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0165709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0165772Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0165986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0166054Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0166282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0166351Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0166579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0166652Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0166879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0166951Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0166955Z 
2025-09-07T09:16:27.0167044Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0167230Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0167288Z     return mod(**inputs)
2025-09-07T09:16:27.0167519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0167586Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0167815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0167879Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0168087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0168153Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0168387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0168780Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0169010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0169116Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0169344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0169463Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0169466Z 
2025-09-07T09:16:27.0169554Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0169740Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0169795Z     return mod(**inputs)
2025-09-07T09:16:27.0170030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0170094Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0170323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0170388Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0170598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0170668Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0170934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0171004Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0171234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0171337Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0171570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0171713Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0171717Z 
2025-09-07T09:16:27.0171807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0171991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0172046Z     return mod(**inputs)
2025-09-07T09:16:27.0172281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0172342Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0172575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0172636Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0172847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0172920Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0173147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0173222Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0173445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0173517Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0173748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0173816Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0173820Z 
2025-09-07T09:16:27.0173912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0174097Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0174192Z     return mod(**inputs)
2025-09-07T09:16:27.0174423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0174485Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0174751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0174815Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0175032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0175099Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0175326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0175396Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0175623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0175698Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0175922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0176021Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0176024Z 
2025-09-07T09:16:27.0176112Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0176294Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0176382Z     return mod(**inputs)
2025-09-07T09:16:27.0176611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0176675Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0176937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0176998Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0177214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0177280Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0177513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0177580Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0177807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0177880Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0178144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0178240Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0178243Z 
2025-09-07T09:16:27.0178330Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0178517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0178572Z     return mod(**inputs)
2025-09-07T09:16:27.0178801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0178866Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0179093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0179158Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0179369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0179434Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0179662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0179731Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0180003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0180075Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0180338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0180434Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0180438Z 
2025-09-07T09:16:27.0180529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0180717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0180771Z     return mod(**inputs)
2025-09-07T09:16:27.0181004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0181068Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0181296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0181362Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0181573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0181646Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0181870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0181972Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0182203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0182276Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0182536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0182604Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0182608Z 
2025-09-07T09:16:27.0182680Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0182770Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0182958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0183019Z     return mod(**inputs)
2025-09-07T09:16:27.0183247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0183316Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0183544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0183606Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0183820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0183885Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0184119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0184201Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0184427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0184515Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0184742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0184810Z     return self.weight * hidden_states
2025-09-07T09:16:27.0184813Z 
2025-09-07T09:16:27.0184901Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0185087Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0185143Z     return mod(**inputs)
2025-09-07T09:16:27.0185409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0185476Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0185705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0185804Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0186015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0186081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0186311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0186388Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0186617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0186724Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0186952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0187025Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0187030Z 
2025-09-07T09:16:27.0187116Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0187298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0187385Z     return mod(**inputs)
2025-09-07T09:16:27.0187622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0187685Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0187958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0188022Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0188236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0188303Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0188527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0188602Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0188830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0188930Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0189159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0189229Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0189232Z 
2025-09-07T09:16:27.0189323Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0189505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0189561Z     return mod(**inputs)
2025-09-07T09:16:27.0189792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0189855Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0190087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0190148Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0190360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0190429Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0190657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0190733Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0191003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0191106Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0191367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0191436Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0191440Z 
2025-09-07T09:16:27.0191511Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0191601Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0191783Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0191838Z     return mod(**inputs)
2025-09-07T09:16:27.0192066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0192131Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0192359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0192420Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0192634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0192698Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0192957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0193029Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0193259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0193358Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0193620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0193691Z     return self.weight * hidden_states
2025-09-07T09:16:27.0193694Z 
2025-09-07T09:16:27.0193780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0193967Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0194023Z     return mod(**inputs)
2025-09-07T09:16:27.0194254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0194317Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0194545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0194610Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0194820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0194891Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0195118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0195186Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0195416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0195486Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0195719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0195785Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0195789Z 
2025-09-07T09:16:27.0195875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0196062Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0196119Z     return mod(**inputs)
2025-09-07T09:16:27.0196350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0196449Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0196678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0196771Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0196983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0197055Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0197283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0197353Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0197579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0197650Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0197881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0197948Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0197953Z 
2025-09-07T09:16:27.0198045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0198225Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0198283Z     return mod(**inputs)
2025-09-07T09:16:27.0198548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0198610Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0198842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0198935Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0199155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0199223Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0199451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0199528Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0199752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0199827Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0200054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0200170Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0200174Z 
2025-09-07T09:16:27.0200263Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0200445Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0200506Z     return mod(**inputs)
2025-09-07T09:16:27.0200735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0200803Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0201032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0201092Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0201308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0201374Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0201603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0201671Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0201896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0202013Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0202237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0202413Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0202417Z 
2025-09-07T09:16:27.0202504Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0202687Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0202747Z     return mod(**inputs)
2025-09-07T09:16:27.0202977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0203182Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0203419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0203483Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0203697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0203770Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0204006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0204078Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0204369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0204445Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0204671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0204800Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0204803Z 
2025-09-07T09:16:27.0204900Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0205090Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0205145Z     return mod(**inputs)
2025-09-07T09:16:27.0205379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0205446Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0205677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0205744Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0205954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0206022Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0206247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0206317Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0206545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0206615Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0206847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0206944Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0206949Z 
2025-09-07T09:16:27.0207038Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0207222Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0207276Z     return mod(**inputs)
2025-09-07T09:16:27.0207514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0207626Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0207860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0207922Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0208186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0208260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0208496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0208571Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0208799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0208871Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0209102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0209203Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0209206Z 
2025-09-07T09:16:27.0209302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0209494Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0209552Z     return mod(**inputs)
2025-09-07T09:16:27.0209793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0209908Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0210149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0210215Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0210466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0210536Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0210768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0210839Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0211067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0211141Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0211365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0211460Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0211463Z 
2025-09-07T09:16:27.0211555Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0211739Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0211799Z     return mod(**inputs)
2025-09-07T09:16:27.0212027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0212093Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0212325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0212391Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0212606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0212677Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0212907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0212975Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0213200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0213312Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0213541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0213610Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0213646Z 
2025-09-07T09:16:27.0213737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0213919Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0213978Z     return mod(**inputs)
2025-09-07T09:16:27.0214209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0214273Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0214501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0214566Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0214776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0214843Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0215072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0215143Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0215376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.0215532Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0215535Z 
2025-09-07T09:16:27.0215605Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0215696Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0215908Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0215969Z     return mod(**inputs)
2025-09-07T09:16:27.0216201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0216265Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0216497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0216557Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0216775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0216841Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0217074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0217142Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0217367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0217470Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0217695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0217768Z     return self.weight * hidden_states
2025-09-07T09:16:27.0217772Z 
2025-09-07T09:16:27.0217857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0218105Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0218167Z     return mod(**inputs)
2025-09-07T09:16:27.0218397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0218461Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0218692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0218753Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0219013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0219079Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0219311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0219417Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0219649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0219723Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0219950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0220022Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0220025Z 
2025-09-07T09:16:27.0220118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0220305Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0220362Z     return mod(**inputs)
2025-09-07T09:16:27.0220593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0220658Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0220887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0220951Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0221197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0221263Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0221496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0221599Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0221835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0221907Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0222135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0222205Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0222208Z 
2025-09-07T09:16:27.0222295Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0222480Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0222537Z     return mod(**inputs)
2025-09-07T09:16:27.0222772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0222833Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0223064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0223130Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0223341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0223409Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0223634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0223702Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0223932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0224005Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0224231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0224350Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0224397Z 
2025-09-07T09:16:27.0224491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0224672Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0224760Z     return mod(**inputs)
2025-09-07T09:16:27.0224995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0225057Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0225292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0225350Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0225558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0225627Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0225854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0225930Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0226154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0226229Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0226456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0226632Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0226636Z 
2025-09-07T09:16:27.0226727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0226909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0226966Z     return mod(**inputs)
2025-09-07T09:16:27.0227226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0227289Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0227526Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0227589Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0227807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0227871Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0228096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0228170Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0228393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0228467Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0228693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0228766Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0228769Z 
2025-09-07T09:16:27.0228857Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0229042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0229101Z     return mod(**inputs)
2025-09-07T09:16:27.0229331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0229399Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0229628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0229690Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0229907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0230006Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0230237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0230356Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0230592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0230669Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0230894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0230992Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0230996Z 
2025-09-07T09:16:27.0231085Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0231272Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0231330Z     return mod(**inputs)
2025-09-07T09:16:27.0231558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0231624Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0231852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0231915Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0232171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0232238Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0232475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0232557Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0232835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0232911Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0233147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0233241Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0233245Z 
2025-09-07T09:16:27.0233332Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0233521Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0233581Z     return mod(**inputs)
2025-09-07T09:16:27.0233815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0233878Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0234110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0234178Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0234389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0234460Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0234688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0234760Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0234991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0235063Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0235291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0235382Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0235385Z 
2025-09-07T09:16:27.0235479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0235703Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0235759Z     return mod(**inputs)
2025-09-07T09:16:27.0235994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0236089Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0236324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0236386Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0236601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0236668Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0236897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0236972Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0237196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0237268Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0237495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0237560Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0237564Z 
2025-09-07T09:16:27.0237669Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0237758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0237946Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0237999Z     return mod(**inputs)
2025-09-07T09:16:27.0238262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0238333Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0238563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0238627Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0238838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0238903Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0239134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0239216Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0239444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0239530Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0239760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0239830Z     return self.weight * hidden_states
2025-09-07T09:16:27.0239834Z 
2025-09-07T09:16:27.0239926Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0240113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0240168Z     return mod(**inputs)
2025-09-07T09:16:27.0240404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0240469Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0240698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0240763Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0240980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0241051Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0241330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0241412Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0241678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0241784Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0242014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0242087Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0242090Z 
2025-09-07T09:16:27.0242183Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0242368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0242423Z     return mod(**inputs)
2025-09-07T09:16:27.0242662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0242725Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0242958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0243020Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0243231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0243343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0243575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0243657Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0243914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0244019Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0244253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0244324Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0244327Z 
2025-09-07T09:16:27.0244423Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0244608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0244666Z     return mod(**inputs)
2025-09-07T09:16:27.0244898Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0244961Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0245194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0245256Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0245473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0245539Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0245765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0245846Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0246071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0246175Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0246439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0246518Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0246523Z 
2025-09-07T09:16:27.0246596Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0246752Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0246939Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0246996Z     return mod(**inputs)
2025-09-07T09:16:27.0247270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0247332Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0247563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0247630Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0247843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0247915Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0248143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0248217Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0248449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0248547Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0248774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0248842Z     return self.weight * hidden_states
2025-09-07T09:16:27.0248846Z 
2025-09-07T09:16:27.0248975Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0249161Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0249216Z     return mod(**inputs)
2025-09-07T09:16:27.0249475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0249540Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0249778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0249838Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0250050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0250121Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0250349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0250426Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0250651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0250722Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0250953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0251024Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0251028Z 
2025-09-07T09:16:27.0251118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0251298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0251361Z     return mod(**inputs)
2025-09-07T09:16:27.0251589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0251651Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0251881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0251943Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0252159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0252223Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0252504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0252580Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0252853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0252925Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0253153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0253222Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0253229Z 
2025-09-07T09:16:27.0253316Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0253497Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0253558Z     return mod(**inputs)
2025-09-07T09:16:27.0253790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0253856Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0254086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0254147Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0254368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0254432Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0254697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0254765Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0255023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0255097Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0255324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0255444Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0255449Z 
2025-09-07T09:16:27.0255536Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0255721Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0255776Z     return mod(**inputs)
2025-09-07T09:16:27.0256005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0256072Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0256302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0256368Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0256579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0256648Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0256879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0256947Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0257174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0257244Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0257469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0257617Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0257621Z 
2025-09-07T09:16:27.0257708Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0257893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0258012Z     return mod(**inputs)
2025-09-07T09:16:27.0258249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0258346Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0258574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0258638Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0258850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0258919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0259147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0259216Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0259444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0259516Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0259746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0259813Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0259817Z 
2025-09-07T09:16:27.0259905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0260126Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0260184Z     return mod(**inputs)
2025-09-07T09:16:27.0260416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0260476Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0260743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0260808Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0261019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0261092Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0261320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0261392Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0261617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0261687Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0261916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0262014Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0262019Z 
2025-09-07T09:16:27.0262113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0262296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0262355Z     return mod(**inputs)
2025-09-07T09:16:27.0262588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0262648Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0262882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0262945Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0263160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0263225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0263454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0263565Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0263791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0263895Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0264121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0264216Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0264224Z 
2025-09-07T09:16:27.0264314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0264498Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0264556Z     return mod(**inputs)
2025-09-07T09:16:27.0264790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0264855Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0265087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0265146Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0265361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0265428Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0265707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0265782Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0266008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0266083Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0266338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0266436Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0266439Z 
2025-09-07T09:16:27.0266526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0266712Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0266772Z     return mod(**inputs)
2025-09-07T09:16:27.0267006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0267070Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0267299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0267364Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0267578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0267643Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0267873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0267940Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0268169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0268238Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0268463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0268536Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0268539Z 
2025-09-07T09:16:27.0268608Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0268700Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0268882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0268975Z     return mod(**inputs)
2025-09-07T09:16:27.0269213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0269275Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0269546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0269608Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0269822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0269892Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0270120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0270193Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0270418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0270517Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0270746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0270814Z     return self.weight * hidden_states
2025-09-07T09:16:27.0270817Z 
2025-09-07T09:16:27.0270909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0271126Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0271189Z     return mod(**inputs)
2025-09-07T09:16:27.0271417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0271482Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0271762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0271824Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0272042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0272109Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0272341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0272409Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0272639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0272717Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0272941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0273016Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0273021Z 
2025-09-07T09:16:27.0273110Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0273296Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0273358Z     return mod(**inputs)
2025-09-07T09:16:27.0273587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0273656Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0273884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0273948Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0274163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0274229Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0274462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0274532Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0274799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0274873Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0275134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0275205Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0275208Z 
2025-09-07T09:16:27.0275298Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0275485Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0275540Z     return mod(**inputs)
2025-09-07T09:16:27.0275772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0275837Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0276069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0276137Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0276348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0276418Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0276648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0276747Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0276981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0277052Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0277320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0277437Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0277442Z 
2025-09-07T09:16:27.0277529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0277717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0277775Z     return mod(**inputs)
2025-09-07T09:16:27.0278009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0278072Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0278300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0278364Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0278578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0278649Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0278873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0278950Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0279178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0279251Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0279481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0279621Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0279624Z 
2025-09-07T09:16:27.0279716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0279897Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0279954Z     return mod(**inputs)
2025-09-07T09:16:27.0280186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0280284Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0280520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0280613Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0280828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0280895Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0281119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0281195Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0281422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0281498Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0281723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0281792Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0281797Z 
2025-09-07T09:16:27.0281890Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0282073Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0282131Z     return mod(**inputs)
2025-09-07T09:16:27.0282393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0282457Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0282692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0282785Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0283001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0283072Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0283302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0283373Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0283599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0283677Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0283900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0284001Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0284004Z 
2025-09-07T09:16:27.0284093Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0284274Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0284337Z     return mod(**inputs)
2025-09-07T09:16:27.0284566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0284634Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0284862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0284922Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0285140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0285206Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0285437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0285507Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0285739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0285844Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0286071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0286202Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0286205Z 
2025-09-07T09:16:27.0286296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0286482Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0286539Z     return mod(**inputs)
2025-09-07T09:16:27.0286770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0286838Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0287068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0287137Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0287349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0287421Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0287654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0287722Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0287984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0288057Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0288289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0288413Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0288416Z 
2025-09-07T09:16:27.0288506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0288700Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0288756Z     return mod(**inputs)
2025-09-07T09:16:27.0288996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0289058Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0289289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0289357Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0289568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0289638Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0289868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0289944Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0290176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0290252Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0290484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0290551Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0290555Z 
2025-09-07T09:16:27.0290652Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0290836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0290894Z     return mod(**inputs)
2025-09-07T09:16:27.0291129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0291190Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0291466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0291528Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0291774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0291849Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0292079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0292155Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0292382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:16:27.0292512Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0292517Z 
2025-09-07T09:16:27.0292589Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0292685Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0292882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0292940Z     return mod(**inputs)
2025-09-07T09:16:27.0293180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0293242Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0293519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0293587Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0293802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0293875Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0294135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0294218Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0294451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0294538Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0294764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0294829Z     return self.weight * hidden_states
2025-09-07T09:16:27.0294833Z 
2025-09-07T09:16:27.0294926Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0295110Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0295168Z     return mod(**inputs)
2025-09-07T09:16:27.0295407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0295471Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0295704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0295764Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0295978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0296049Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0296275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0296357Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0296584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0296694Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0296924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0297053Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0297056Z 
2025-09-07T09:16:27.0297152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0297375Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0297435Z     return mod(**inputs)
2025-09-07T09:16:27.0297667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0297731Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0298010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0298073Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0298295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0298367Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0298596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0298674Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0298902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0299005Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0299271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0299351Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0299354Z 
2025-09-07T09:16:27.0299442Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0299655Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0299717Z     return mod(**inputs)
2025-09-07T09:16:27.0299952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0300018Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0300251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0300312Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0300529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0300593Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0300822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0300899Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0301125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0301232Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0301456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0301529Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0301532Z 
2025-09-07T09:16:27.0301601Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0301692Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0301875Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0301930Z     return mod(**inputs)
2025-09-07T09:16:27.0302164Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0302226Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0302460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0302557Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0302766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0302869Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0303229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0303305Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0303535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0303633Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0303866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0303934Z     return self.weight * hidden_states
2025-09-07T09:16:27.0303938Z 
2025-09-07T09:16:27.0304033Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0304217Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0304280Z     return mod(**inputs)
2025-09-07T09:16:27.0304513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0304577Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0304886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0304949Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0305165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0305235Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0305507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0305586Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0305816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0305896Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0306122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0306194Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0306198Z 
2025-09-07T09:16:27.0306290Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0306471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0306533Z     return mod(**inputs)
2025-09-07T09:16:27.0306764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0306830Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0307060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0307120Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0307336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0307405Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0307637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0307704Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0307932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0308007Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0308236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0308356Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0308359Z 
2025-09-07T09:16:27.0308445Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0308635Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0308740Z     return mod(**inputs)
2025-09-07T09:16:27.0308971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0309042Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0309274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0309339Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0309553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0309617Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0309848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0309918Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0310149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0310222Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0310484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0310610Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0310613Z 
2025-09-07T09:16:27.0310699Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0310883Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0310971Z     return mod(**inputs)
2025-09-07T09:16:27.0311208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0311273Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0311502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0311572Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0311782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0311852Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0312079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0319216Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0319527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0319610Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0319862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0320017Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0320023Z 
2025-09-07T09:16:27.0320125Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0320336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0320396Z     return mod(**inputs)
2025-09-07T09:16:27.0320654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0320722Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0320956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0321026Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0321249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0321407Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0321642Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0321763Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0321995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0322070Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0322301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0322373Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0322376Z 
2025-09-07T09:16:27.0322474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0322685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0322747Z     return mod(**inputs)
2025-09-07T09:16:27.0322988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0323053Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0323285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0323348Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0323597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0323673Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0323901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0324008Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0324236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0324313Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0324548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0324652Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0324656Z 
2025-09-07T09:16:27.0324758Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0324950Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0325012Z     return mod(**inputs)
2025-09-07T09:16:27.0325249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0325311Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0325550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0325611Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0325834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0325905Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0326134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0326213Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0326443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0326520Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0326748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0326842Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0326888Z 
2025-09-07T09:16:27.0326979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0327164Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0327229Z     return mod(**inputs)
2025-09-07T09:16:27.0327494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0327563Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0327793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0327856Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0330589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0330670Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0330921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0331007Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0331242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0331326Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0331555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0331659Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0331732Z 
2025-09-07T09:16:27.0331838Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0332049Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0332140Z     return mod(**inputs)
2025-09-07T09:16:27.0332422Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0332497Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0332733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0332805Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0333023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0333104Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0333340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0333415Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0333648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0333725Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0333961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0334032Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0334035Z 
2025-09-07T09:16:27.0334110Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0334213Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0334406Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0334473Z     return mod(**inputs)
2025-09-07T09:16:27.0334718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0334790Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0335037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0335105Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0335335Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0335443Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0335686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0335761Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0335989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0336097Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0336337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0336496Z     return self.weight * hidden_states
2025-09-07T09:16:27.0336499Z 
2025-09-07T09:16:27.0336599Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0336798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0336865Z     return mod(**inputs)
2025-09-07T09:16:27.0337100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0337172Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0337399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0337462Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0337719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0337795Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0338108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0338178Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0338442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0338517Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0338744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0338815Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0338819Z 
2025-09-07T09:16:27.0338911Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0339104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0339161Z     return mod(**inputs)
2025-09-07T09:16:27.0339392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0339458Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0339687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0339750Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0339975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0340044Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0340283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0340352Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0340585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0340657Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0340884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0340952Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0340956Z 
2025-09-07T09:16:27.0341051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0341286Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0341344Z     return mod(**inputs)
2025-09-07T09:16:27.0341576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0341637Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0341867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0341928Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0342144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0342260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0342491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0342563Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0342787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0342859Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0343084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0343208Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0343211Z 
2025-09-07T09:16:27.0343360Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0343553Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0343614Z     return mod(**inputs)
2025-09-07T09:16:27.0343845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0343957Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0344190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0344253Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0344469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0344539Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0344767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0344838Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0345062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0345135Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0345361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0345515Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0345519Z 
2025-09-07T09:16:27.0345609Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0345802Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0345860Z     return mod(**inputs)
2025-09-07T09:16:27.0346092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0346160Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0346389Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0346456Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0346671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0346736Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0347011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0347083Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0347315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0347386Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0347616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0347687Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0347691Z 
2025-09-07T09:16:27.0347779Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0348007Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0348064Z     return mod(**inputs)
2025-09-07T09:16:27.0348299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0348363Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0348592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0348657Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0348868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0348941Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0349200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0349273Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0349537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0349610Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0349844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0349944Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0349948Z 
2025-09-07T09:16:27.0350041Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0350228Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0350284Z     return mod(**inputs)
2025-09-07T09:16:27.0350521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0350583Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0350816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0350881Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0351093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0351166Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0351394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0351469Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0351693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0351766Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0351998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0352092Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0352095Z 
2025-09-07T09:16:27.0352187Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0352370Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0352464Z     return mod(**inputs)
2025-09-07T09:16:27.0352694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0352759Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0352994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0353055Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0353273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0353342Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0353610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0353685Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0353910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0353986Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0354209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0354301Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0354308Z 
2025-09-07T09:16:27.0354396Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0354609Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0354673Z     return mod(**inputs)
2025-09-07T09:16:27.0354904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0354971Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0355231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0355296Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0355513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0355578Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0355810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0355876Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0356104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0356180Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0356405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0356479Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0356482Z 
2025-09-07T09:16:27.0356556Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0356648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0356829Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0356887Z     return mod(**inputs)
2025-09-07T09:16:27.0357120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0357182Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0357416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0357479Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0357690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0357759Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0357987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0358113Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0358340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0358426Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0358658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0358725Z     return self.weight * hidden_states
2025-09-07T09:16:27.0358729Z 
2025-09-07T09:16:27.0358822Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0359044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0359103Z     return mod(**inputs)
2025-09-07T09:16:27.0359337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0359398Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0359633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0359695Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0359910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0359974Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0360237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0360320Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0360547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0360692Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0360924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0360995Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0361003Z 
2025-09-07T09:16:27.0361090Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0361278Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0361338Z     return mod(**inputs)
2025-09-07T09:16:27.0361567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0361631Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0361861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0361922Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0362143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0362209Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0362442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0362519Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0362748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0362850Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0363081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0363155Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0363159Z 
2025-09-07T09:16:27.0363244Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0363430Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0363520Z     return mod(**inputs)
2025-09-07T09:16:27.0363755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0363816Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0364048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0364109Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0364323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0364392Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0364659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0364740Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0364970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0365073Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0365303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0365370Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0365374Z 
2025-09-07T09:16:27.0365467Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0365696Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0365755Z     return mod(**inputs)
2025-09-07T09:16:27.0365984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0366049Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0366316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0366379Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0366592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0366658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0366884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0366963Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0367189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:16:27.0367311Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:16:27.0367316Z 
2025-09-07T09:16:27.0367385Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0367476Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0367659Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0367714Z     return mod(**inputs)
2025-09-07T09:16:27.0367946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0368009Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0368238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0368295Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0368511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0368580Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0368805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0368881Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0369105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0369235Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0369465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0369530Z     return self.weight * hidden_states
2025-09-07T09:16:27.0369534Z 
2025-09-07T09:16:27.0369622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0369804Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0369859Z     return mod(**inputs)
2025-09-07T09:16:27.0370092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0370188Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0370421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0370484Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0370701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0370765Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0370990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0371062Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0371320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0371395Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0371626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0371725Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0371729Z 
2025-09-07T09:16:27.0371821Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0372005Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0372065Z     return mod(**inputs)
2025-09-07T09:16:27.0372293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0372354Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0372589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0372651Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0372863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0372928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0373157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0373229Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0373453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0373525Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0373750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0373819Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0373822Z 
2025-09-07T09:16:27.0373909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0374091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0374151Z     return mod(**inputs)
2025-09-07T09:16:27.0374380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0374446Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0374714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0374775Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0374991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0375056Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0375286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0375359Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0375585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0375693Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0375921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0376046Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0376050Z 
2025-09-07T09:16:27.0376139Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0376325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0376380Z     return mod(**inputs)
2025-09-07T09:16:27.0376610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0376712Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0376947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0377017Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0377267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0377337Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0377569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0377638Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0377863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0377933Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0378210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0378357Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0378362Z 
2025-09-07T09:16:27.0378456Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0378649Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0378704Z     return mod(**inputs)
2025-09-07T09:16:27.0378938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0379001Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0379227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0379291Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0379500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0379571Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0379798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0379873Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0380101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0380172Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0380445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0380512Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0380515Z 
2025-09-07T09:16:27.0380608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0380792Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0380845Z     return mod(**inputs)
2025-09-07T09:16:27.0381076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0381136Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0381416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0381479Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0381691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0381761Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0381988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0382061Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0382285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0382391Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0382622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0382722Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0382726Z 
2025-09-07T09:16:27.0382848Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0383032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0383096Z     return mod(**inputs)
2025-09-07T09:16:27.0383328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0383392Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0383623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0383685Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0383901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0383966Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0384195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0384267Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0384492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0384566Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0384794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0384890Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0384893Z 
2025-09-07T09:16:27.0384980Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0385164Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0385222Z     return mod(**inputs)
2025-09-07T09:16:27.0385451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0385516Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0385746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0385839Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0386056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0386123Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0386350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0386416Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0386646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0386765Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0386991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0387086Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0387091Z 
2025-09-07T09:16:27.0387177Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0387360Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0387413Z     return mod(**inputs)
2025-09-07T09:16:27.0387640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0387703Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0387962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0388024Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0388240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0388304Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0388563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0388633Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0388861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0388928Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0389158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0389223Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0389228Z 
2025-09-07T09:16:27.0389297Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0389386Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0389568Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0389623Z     return mod(**inputs)
2025-09-07T09:16:27.0389851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0389912Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0390141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0390200Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0390412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0390475Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0390700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0390771Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0390996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0391092Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0391318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0391761Z     return self.weight * hidden_states
2025-09-07T09:16:27.0391764Z 
2025-09-07T09:16:27.0391858Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0392045Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0392104Z     return mod(**inputs)
2025-09-07T09:16:27.0392340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0392405Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0392631Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0392732Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0392952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0393023Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0393252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0393320Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0393547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0393618Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0393877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0393947Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0393951Z 
2025-09-07T09:16:27.0394040Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0394256Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0394313Z     return mod(**inputs)
2025-09-07T09:16:27.0394542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0394606Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0394835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0394901Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0395113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0395178Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0395405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0395474Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0395703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0395774Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0395999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0396064Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0396067Z 
2025-09-07T09:16:27.0396154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0396337Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0396393Z     return mod(**inputs)
2025-09-07T09:16:27.0396621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0396683Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0396911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0396973Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0397230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0397300Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0397525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0397592Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0397820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0397889Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0398116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0398275Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0398280Z 
2025-09-07T09:16:27.0398368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0398550Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0398605Z     return mod(**inputs)
2025-09-07T09:16:27.0398835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0398897Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0399129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0399222Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0399436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0399505Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0399764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0399837Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0400062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0400132Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0400358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0400502Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0400505Z 
2025-09-07T09:16:27.0400596Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0400777Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0400834Z     return mod(**inputs)
2025-09-07T09:16:27.0401064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0401124Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0401357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0401417Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0401628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0401691Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0401918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0401990Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0402213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0402292Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0402517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0402621Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0402624Z 
2025-09-07T09:16:27.0402712Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0402893Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0402950Z     return mod(**inputs)
2025-09-07T09:16:27.0403364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0403429Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0403656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0403799Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0404013Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0404077Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0404304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0404370Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0404594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0404665Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0404936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0405038Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0405042Z 
2025-09-07T09:16:27.0405131Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0405315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0405411Z     return mod(**inputs)
2025-09-07T09:16:27.0405644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0405708Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0405937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0405997Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0406208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0406272Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0406501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0406569Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0406798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0406867Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0407092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0407186Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0407189Z 
2025-09-07T09:16:27.0407275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0407457Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0407510Z     return mod(**inputs)
2025-09-07T09:16:27.0407743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0407804Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0408034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0408095Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0408306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0408437Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0408663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0408731Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0408956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0409026Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0409253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0409387Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0409390Z 
2025-09-07T09:16:27.0409479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0409660Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0409718Z     return mod(**inputs)
2025-09-07T09:16:27.0409950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0410010Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0410244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0410304Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0410554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0410623Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0410851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0410957Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0411185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0411260Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0411496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0411562Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0411565Z 
2025-09-07T09:16:27.0411638Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0411727Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0411918Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0411974Z     return mod(**inputs)
2025-09-07T09:16:27.0412203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0412270Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0412501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0412566Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0412776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0412842Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0413072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0413153Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0413382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0413469Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0413694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0413766Z     return self.weight * hidden_states
2025-09-07T09:16:27.0413803Z 
2025-09-07T09:16:27.0413893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0414079Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0414132Z     return mod(**inputs)
2025-09-07T09:16:27.0414365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0414424Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0414657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0414720Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0414977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0415050Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0415285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0415369Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0415599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0415707Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0415937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0416039Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0416043Z 
2025-09-07T09:16:27.0416134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0416318Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0416374Z     return mod(**inputs)
2025-09-07T09:16:27.0416637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0416701Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0416931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0416990Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0417200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0417270Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0417499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0417575Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0417802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0417906Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0418184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0418256Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0418260Z 
2025-09-07T09:16:27.0418352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0418534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0418590Z     return mod(**inputs)
2025-09-07T09:16:27.0418820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0418882Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0419116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0419175Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0419392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0419496Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0419725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0419799Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0420028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0420132Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0420361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0420469Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0420476Z 
2025-09-07T09:16:27.0420545Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0420634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0420817Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0420874Z     return mod(**inputs)
2025-09-07T09:16:27.0421105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0421166Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0421397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0421459Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0421702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0421771Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0422028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0422100Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0422332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0422427Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0422656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0422720Z     return self.weight * hidden_states
2025-09-07T09:16:27.0422723Z 
2025-09-07T09:16:27.0422812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0422995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0423049Z     return mod(**inputs)
2025-09-07T09:16:27.0423281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0423343Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0423573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0423632Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0423842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0423909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0424134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0424204Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0424432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0424506Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0424734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0424801Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0424840Z 
2025-09-07T09:16:27.0424935Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0425115Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0425171Z     return mod(**inputs)
2025-09-07T09:16:27.0425401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0425461Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0425693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0425752Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0426017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0426082Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0426311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0426384Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0426609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0426682Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0426904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0426971Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0427006Z 
2025-09-07T09:16:27.0427096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0427278Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0427343Z     return mod(**inputs)
2025-09-07T09:16:27.0427622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0427689Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0427924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0427985Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0428197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0428261Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0428490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0428558Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0428785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0428860Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0429087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0429210Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0429214Z 
2025-09-07T09:16:27.0429302Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0429484Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0429544Z     return mod(**inputs)
2025-09-07T09:16:27.0429774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0429839Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0430067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0430130Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0430343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0430442Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0430673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0430741Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0430976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0431043Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0431271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0431417Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0431479Z 
2025-09-07T09:16:27.0431569Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0431757Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0431811Z     return mod(**inputs)
2025-09-07T09:16:27.0432046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0432109Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0432337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0432403Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0432645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0432711Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0432940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0433012Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0433271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0433342Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0433572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0433640Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0433643Z 
2025-09-07T09:16:27.0433732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0433918Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0433975Z     return mod(**inputs)
2025-09-07T09:16:27.0434209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0434270Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0434501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0434561Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0434771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0434841Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0435067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0435135Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0435361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0435431Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0435659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0435757Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0435760Z 
2025-09-07T09:16:27.0435850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0436072Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0436127Z     return mod(**inputs)
2025-09-07T09:16:27.0436359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0436420Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0436648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0436712Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0436923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0437020Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0437247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0437317Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0437547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0437617Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0437843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0437932Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0437936Z 
2025-09-07T09:16:27.0438057Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0438242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0438305Z     return mod(**inputs)
2025-09-07T09:16:27.0438535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0438628Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0438862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0438924Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0439137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0439202Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0439431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0439501Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0439725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0439811Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0440038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0440129Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0440134Z 
2025-09-07T09:16:27.0440222Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0440403Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0440459Z     return mod(**inputs)
2025-09-07T09:16:27.0440688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0440752Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0440983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0441044Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0441254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0441320Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0441549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0441651Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0441880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0441949Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0442180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0442251Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0442255Z 
2025-09-07T09:16:27.0442348Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0442577Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0442634Z     return mod(**inputs)
2025-09-07T09:16:27.0442873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0442943Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0443171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0443236Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0443453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0443526Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0443787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0443859Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0444092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.0444245Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0444248Z 
2025-09-07T09:16:27.0444325Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0444415Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0444601Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0444658Z     return mod(**inputs)
2025-09-07T09:16:27.0444889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0444955Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0445185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0445249Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0445465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0445533Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0445763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0445835Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0446062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0446159Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0446385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0446453Z     return self.weight * hidden_states
2025-09-07T09:16:27.0446456Z 
2025-09-07T09:16:27.0446544Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0446730Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0446786Z     return mod(**inputs)
2025-09-07T09:16:27.0447017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0447116Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0447349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0447414Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0447627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0447695Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0447931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0448001Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0448267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0448341Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0448575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0448644Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0448647Z 
2025-09-07T09:16:27.0448740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0448930Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0448986Z     return mod(**inputs)
2025-09-07T09:16:27.0449264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0449328Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0449559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0449624Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0449868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0449942Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0450168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0450239Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0450471Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0450541Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0450773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0450840Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0450845Z 
2025-09-07T09:16:27.0450938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0451120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0451175Z     return mod(**inputs)
2025-09-07T09:16:27.0451409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0451468Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0451701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0451759Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0451972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0452041Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0452265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0452340Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0452566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0452673Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0452900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0453021Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0453025Z 
2025-09-07T09:16:27.0453118Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0453302Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0453363Z     return mod(**inputs)
2025-09-07T09:16:27.0453593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0453689Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0453921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0453982Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0454197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0454260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0454487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0454559Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0454813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0454890Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0455115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0455295Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0455298Z 
2025-09-07T09:16:27.0455390Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0455575Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0455632Z     return mod(**inputs)
2025-09-07T09:16:27.0455862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0455927Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0456158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0456219Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0456437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0456503Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0456733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0456804Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0457037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0457107Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0457331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0457401Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0457404Z 
2025-09-07T09:16:27.0457494Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0457682Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0457740Z     return mod(**inputs)
2025-09-07T09:16:27.0458007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0458073Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0458345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0458409Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0458621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0458687Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0458917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0458987Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0459217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0459323Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0459555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0459657Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0459660Z 
2025-09-07T09:16:27.0459749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0459933Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0459988Z     return mod(**inputs)
2025-09-07T09:16:27.0460221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0460317Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0460552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0460620Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0460864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0460934Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0461162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0461232Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0461467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0461536Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0461768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0461858Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0461861Z 
2025-09-07T09:16:27.0461954Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0462135Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0462191Z     return mod(**inputs)
2025-09-07T09:16:27.0462425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0462488Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0462721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0462783Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0462994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0463065Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0463291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0463366Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0463592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0463663Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0463928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0464024Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0464027Z 
2025-09-07T09:16:27.0464120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0464304Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0464363Z     return mod(**inputs)
2025-09-07T09:16:27.0464595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0464657Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0464928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0464990Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0465205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0465272Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0465498Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0465571Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0465796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0465898Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0466126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0466202Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0466206Z 
2025-09-07T09:16:27.0466278Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0466397Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0466585Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0466642Z     return mod(**inputs)
2025-09-07T09:16:27.0466877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0466939Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0467167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0467233Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0467443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0467513Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0467743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0467825Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0468056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0468143Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0468378Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0468443Z     return self.weight * hidden_states
2025-09-07T09:16:27.0468447Z 
2025-09-07T09:16:27.0468542Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0468729Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0468787Z     return mod(**inputs)
2025-09-07T09:16:27.0469027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0469091Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0469324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0469424Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0469640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0469712Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0469940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0470025Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0470253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0470414Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0470649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0470721Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0470725Z 
2025-09-07T09:16:27.0470817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0471002Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0471063Z     return mod(**inputs)
2025-09-07T09:16:27.0471293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0471355Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0471622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0471685Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0471908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0472009Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0472243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0472330Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0472555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0472662Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0472887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0472970Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0472974Z 
2025-09-07T09:16:27.0473062Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0473245Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0473302Z     return mod(**inputs)
2025-09-07T09:16:27.0473535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0473603Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0473834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0473894Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0474109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0474177Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0474409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0474484Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0474708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0474815Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0475076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0475152Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0475155Z 
2025-09-07T09:16:27.0475228Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0475318Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0475501Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0475561Z     return mod(**inputs)
2025-09-07T09:16:27.0475800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0475894Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0476128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0476187Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0476402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0476477Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0476707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0476783Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0477007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0477136Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0477374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0477444Z     return self.weight * hidden_states
2025-09-07T09:16:27.0477447Z 
2025-09-07T09:16:27.0477575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0477763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0477831Z     return mod(**inputs)
2025-09-07T09:16:27.0478061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0478122Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0478359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0478424Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0478644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0478713Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0478942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0479022Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0479251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0479328Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0479551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0479621Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0479629Z 
2025-09-07T09:16:27.0479721Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0479908Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0479973Z     return mod(**inputs)
2025-09-07T09:16:27.0480203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0480274Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0480503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0480601Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0480818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0480886Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0481118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0481186Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0481413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0481528Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0481759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0481836Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0481839Z 
2025-09-07T09:16:27.0481927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0482112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0482169Z     return mod(**inputs)
2025-09-07T09:16:27.0482398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0482464Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0482739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0482807Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0483020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0483089Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0483367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0483440Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0483671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0483743Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0483967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0484088Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0484093Z 
2025-09-07T09:16:27.0484181Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0484368Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0484425Z     return mod(**inputs)
2025-09-07T09:16:27.0484660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0484722Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0484948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0485017Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0485227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0485293Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0485522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0485592Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0485824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0485893Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0486125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0486321Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0486324Z 
2025-09-07T09:16:27.0486414Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0486596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0486652Z     return mod(**inputs)
2025-09-07T09:16:27.0486886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0486949Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0487180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0487277Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0487489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0487563Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0487797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0487873Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0488097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0488170Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0488436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0488512Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0488517Z 
2025-09-07T09:16:27.0488613Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0488826Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0488888Z     return mod(**inputs)
2025-09-07T09:16:27.0489123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0489186Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0489423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0489483Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0489701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0489769Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0489999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0490075Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0490302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0490379Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0490603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0490703Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0490712Z 
2025-09-07T09:16:27.0490800Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0490984Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0491047Z     return mod(**inputs)
2025-09-07T09:16:27.0491276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0491345Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0491574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0491639Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0491913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0491977Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0492212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0492283Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0492509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0492587Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0492811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0492941Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0492946Z 
2025-09-07T09:16:27.0493039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0493227Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0493283Z     return mod(**inputs)
2025-09-07T09:16:27.0493510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0493579Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0493806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0493918Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0494135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0494204Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0494473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0494546Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0494780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0494849Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0495076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0495173Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0495177Z 
2025-09-07T09:16:27.0495267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0495454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0495513Z     return mod(**inputs)
2025-09-07T09:16:27.0495746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0495812Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0496039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0496108Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0496321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0496395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0496623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0496695Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0496925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0496993Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0497225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0497292Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0497327Z 
2025-09-07T09:16:27.0497401Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0497495Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0497676Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0497736Z     return mod(**inputs)
2025-09-07T09:16:27.0498005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0498079Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0498312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0498409Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0498630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0498697Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0498931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0499003Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0499228Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0499324Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0499581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0499651Z     return self.weight * hidden_states
2025-09-07T09:16:27.0499654Z 
2025-09-07T09:16:27.0499743Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0499926Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0500018Z     return mod(**inputs)
2025-09-07T09:16:27.0500252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0500318Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0500548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0500613Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0500826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0500892Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0501121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0501192Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0501425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0501497Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0501725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0501798Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0501801Z 
2025-09-07T09:16:27.0501891Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0502074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0502129Z     return mod(**inputs)
2025-09-07T09:16:27.0502358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0502423Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0502654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0502723Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0502936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0503193Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0503427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0503500Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0503734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0503811Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0504043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0504180Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0504183Z 
2025-09-07T09:16:27.0504276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0504473Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0504537Z     return mod(**inputs)
2025-09-07T09:16:27.0504773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0504838Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0505067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0505137Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0505395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0505471Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0505701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0505818Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0506058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0506136Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0506372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0506491Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0506494Z 
2025-09-07T09:16:27.0506593Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0506781Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0506842Z     return mod(**inputs)
2025-09-07T09:16:27.0507079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0507145Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0507379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0507442Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0507657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0507730Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0507959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0508033Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0508260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0508334Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0508561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0508708Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0508756Z 
2025-09-07T09:16:27.0508850Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0509036Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0509097Z     return mod(**inputs)
2025-09-07T09:16:27.0509326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0509390Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0509625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0509685Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0509941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0510007Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0510241Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0510311Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0510537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0510612Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0510838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0510913Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0510949Z 
2025-09-07T09:16:27.0511039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0511221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0511284Z     return mod(**inputs)
2025-09-07T09:16:27.0511563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0511631Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0511864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0511926Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0512143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0512207Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0512438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0512506Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0512737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0512809Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0513034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0513138Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0513142Z 
2025-09-07T09:16:27.0513235Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0513420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0513476Z     return mod(**inputs)
2025-09-07T09:16:27.0513708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0513776Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0514022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0514088Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0514303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0514372Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0514639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0514711Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0514944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0515015Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0515248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0515343Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0515387Z 
2025-09-07T09:16:27.0515479Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0515670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0515725Z     return mod(**inputs)
2025-09-07T09:16:27.0515964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0516029Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0516261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0516331Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0516543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0516651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0516884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0516964Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0517225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0517299Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0517535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0517628Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0517632Z 
2025-09-07T09:16:27.0517728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0517910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0517967Z     return mod(**inputs)
2025-09-07T09:16:27.0518205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0518272Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0518512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0518575Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0518792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0518869Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0519097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0519170Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0519399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0519481Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0519711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0519781Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0519785Z 
2025-09-07T09:16:27.0519883Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0520066Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0520194Z     return mod(**inputs)
2025-09-07T09:16:27.0520424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0520490Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0520725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0520786Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0521006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0521110Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0521339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0521414Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0521640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:16:27.0521768Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0521771Z 
2025-09-07T09:16:27.0521841Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0521937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0522120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0522212Z     return mod(**inputs)
2025-09-07T09:16:27.0522446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0522512Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0522777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0522839Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0523052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0523125Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0523353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0523440Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0523666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0523759Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0523985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0524056Z     return self.weight * hidden_states
2025-09-07T09:16:27.0524059Z 
2025-09-07T09:16:27.0524159Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0524344Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0524403Z     return mod(**inputs)
2025-09-07T09:16:27.0524632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0524694Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0524926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0524989Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0525204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0525272Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0525500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0525589Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0525857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0525969Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0526204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0526281Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0526284Z 
2025-09-07T09:16:27.0526375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0526558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0526652Z     return mod(**inputs)
2025-09-07T09:16:27.0526885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0526953Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0527181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0527245Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0527462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0527530Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0527758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0527871Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0528103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0528213Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0528466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0528546Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0528549Z 
2025-09-07T09:16:27.0528639Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0528828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0528886Z     return mod(**inputs)
2025-09-07T09:16:27.0529116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0529182Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0529417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0529483Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0529694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0529760Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0529990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0530069Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0530299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0530400Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0530633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0530705Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0530708Z 
2025-09-07T09:16:27.0530779Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0530873Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0531060Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0531120Z     return mod(**inputs)
2025-09-07T09:16:27.0531387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0531451Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0531687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0531750Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0531962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0532029Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0532253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0532359Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0532593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0532691Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0532915Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0532983Z     return self.weight * hidden_states
2025-09-07T09:16:27.0532986Z 
2025-09-07T09:16:27.0533075Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0533255Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0533312Z     return mod(**inputs)
2025-09-07T09:16:27.0533827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0533903Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0534165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0541572Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0541863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0541943Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0542206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0542288Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0542521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0542607Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0542845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0542919Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0542928Z 
2025-09-07T09:16:27.0543037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0543239Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0543308Z     return mod(**inputs)
2025-09-07T09:16:27.0543548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0543622Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0543856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0543922Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0544152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0544227Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0544465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0544542Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0544770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0544921Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0545149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0545223Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0545227Z 
2025-09-07T09:16:27.0545328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0545535Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0545595Z     return mod(**inputs)
2025-09-07T09:16:27.0545883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0545962Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0546212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0546284Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0546503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0546572Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0546820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0546896Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0547174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0547252Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0547488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0547646Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0547652Z 
2025-09-07T09:16:27.0547750Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0547956Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0548016Z     return mod(**inputs)
2025-09-07T09:16:27.0548253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0548321Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0548553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0548620Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0548839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0548917Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0549151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0549225Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0549456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0549529Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0549760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0549910Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0549914Z 
2025-09-07T09:16:27.0550011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0550203Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0550262Z     return mod(**inputs)
2025-09-07T09:16:27.0550496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0550593Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0550833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0550897Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0551114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0551183Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0551414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0551491Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0551754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0551828Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0552059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0552130Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0552134Z 
2025-09-07T09:16:27.0552231Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0552415Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0552477Z     return mod(**inputs)
2025-09-07T09:16:27.0552760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0552832Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0553084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0553153Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0553413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0553489Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0553725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0553805Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0554037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0554121Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0554349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0554450Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0554460Z 
2025-09-07T09:16:27.0554560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0554763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0554826Z     return mod(**inputs)
2025-09-07T09:16:27.0555060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0555134Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0555365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0555429Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0555648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0555719Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0555954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0556026Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0556253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0556364Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0556590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0556688Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0556691Z 
2025-09-07T09:16:27.0556783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0556973Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0557035Z     return mod(**inputs)
2025-09-07T09:16:27.0557268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0557372Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0557608Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0557677Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0557892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0557961Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0558193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0558263Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0558525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0558596Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0558827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0558924Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0558958Z 
2025-09-07T09:16:27.0559051Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0559243Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0559300Z     return mod(**inputs)
2025-09-07T09:16:27.0559539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0559606Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0559835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0559902Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0560118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0560191Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0560419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0560488Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0560727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0560795Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0561029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0561099Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0561102Z 
2025-09-07T09:16:27.0561182Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0561274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0561456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0561520Z     return mod(**inputs)
2025-09-07T09:16:27.0561753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0561821Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0562087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0562151Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0562370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0562439Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0562673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0562745Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0562972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0563123Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0563352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0563428Z     return self.weight * hidden_states
2025-09-07T09:16:27.0563431Z 
2025-09-07T09:16:27.0563524Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0563714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0563771Z     return mod(**inputs)
2025-09-07T09:16:27.0564001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0564104Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0564336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0564406Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0564648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0564718Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0564955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0565023Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0565253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0565326Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0565554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0565630Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0565634Z 
2025-09-07T09:16:27.0565722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0565914Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0565974Z     return mod(**inputs)
2025-09-07T09:16:27.0566210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0566276Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0566505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0566572Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0566786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0566858Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0567086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0567159Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0567401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0567475Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0567744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0567813Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0567816Z 
2025-09-07T09:16:27.0567912Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0568096Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0568152Z     return mod(**inputs)
2025-09-07T09:16:27.0568388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0568450Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0568720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0568787Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0569001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0569076Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0569302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0569378Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0569605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0569709Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0569945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0570072Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0570076Z 
2025-09-07T09:16:27.0570205Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0570392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0570457Z     return mod(**inputs)
2025-09-07T09:16:27.0570692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0570754Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0570989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0571052Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0571269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0571337Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0571568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0571646Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0571871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0571952Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0572179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0572329Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0572338Z 
2025-09-07T09:16:27.0572432Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0572616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0572679Z     return mod(**inputs)
2025-09-07T09:16:27.0572911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0572980Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0573208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0573306Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0573524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0573589Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0573825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0573896Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0574127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0574244Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0574472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0574548Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0574551Z 
2025-09-07T09:16:27.0574643Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0574833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0574892Z     return mod(**inputs)
2025-09-07T09:16:27.0575122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0575190Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0575454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0575521Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0575736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0575804Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0576069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0576142Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0576374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0576447Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0576676Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0576782Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0576787Z 
2025-09-07T09:16:27.0576879Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0577071Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0577130Z     return mod(**inputs)
2025-09-07T09:16:27.0577371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0577433Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0577666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0577734Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0577948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0578067Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0578300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0578372Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0578606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0578679Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0578918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0579052Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0579055Z 
2025-09-07T09:16:27.0579144Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0579334Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0579394Z     return mod(**inputs)
2025-09-07T09:16:27.0579630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0579694Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0579929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0580027Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0580239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0580314Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0580542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0580619Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0580846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0580918Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0581183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0581279Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0581283Z 
2025-09-07T09:16:27.0581377Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0581589Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0581653Z     return mod(**inputs)
2025-09-07T09:16:27.0581888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0581952Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0582186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0582247Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0582465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0582530Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0582761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0582831Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0583063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0583137Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0583361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0583436Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0583439Z 
2025-09-07T09:16:27.0583511Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0583606Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0583792Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0583848Z     return mod(**inputs)
2025-09-07T09:16:27.0584082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0584147Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0584379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0584490Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0584705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0584772Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0584997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0585081Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0585307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0585394Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0585660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0585728Z     return self.weight * hidden_states
2025-09-07T09:16:27.0585732Z 
2025-09-07T09:16:27.0585824Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0586006Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0586066Z     return mod(**inputs)
2025-09-07T09:16:27.0586295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0586359Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0586629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0586691Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0586907Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0586975Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0587245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0587325Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0587552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0587666Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0587893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0587968Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0587972Z 
2025-09-07T09:16:27.0588059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0588241Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0588300Z     return mod(**inputs)
2025-09-07T09:16:27.0588528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0588597Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0588825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0588889Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0589101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0589167Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0589399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0589476Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0589704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0589807Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0590033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0590154Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0590157Z 
2025-09-07T09:16:27.0590244Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0590434Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0590491Z     return mod(**inputs)
2025-09-07T09:16:27.0590723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0590789Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0591017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0591114Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0591327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0591400Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0591627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0591701Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0591932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0592033Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0592301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0592370Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0592375Z 
2025-09-07T09:16:27.0592464Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0592680Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0592738Z     return mod(**inputs)
2025-09-07T09:16:27.0592972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0593035Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0593263Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0593326Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0593537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0593608Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0593835Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0593914Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0594141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:16:27.0594259Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:16:27.0594264Z 
2025-09-07T09:16:27.0594337Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0594425Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0594607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0594661Z     return mod(**inputs)
2025-09-07T09:16:27.0594888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0594956Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0595184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0595250Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0595463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0595529Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0595797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0595869Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0596100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0596196Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0596427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0596496Z     return self.weight * hidden_states
2025-09-07T09:16:27.0596533Z 
2025-09-07T09:16:27.0596627Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0596820Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0596878Z     return mod(**inputs)
2025-09-07T09:16:27.0597113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0597175Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0597405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0597472Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0597683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0597788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0598020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0598098Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0598363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0598442Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0598670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0598737Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0598740Z 
2025-09-07T09:16:27.0598834Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0599014Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0599071Z     return mod(**inputs)
2025-09-07T09:16:27.0599304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0599367Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0599600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0599662Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0599871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0599946Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0600173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0600247Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0600476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0600557Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0600782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0600852Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0600855Z 
2025-09-07T09:16:27.0600952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0601133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0601230Z     return mod(**inputs)
2025-09-07T09:16:27.0601461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0601522Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0601754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0601814Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0602031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0602096Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0602358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0602433Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0602661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0602738Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0602962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0603275Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0603279Z 
2025-09-07T09:16:27.0603368Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0603633Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0603693Z     return mod(**inputs)
2025-09-07T09:16:27.0603927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0603996Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0604298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0604364Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0604582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0604652Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0604882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0604951Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0605175Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0605248Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0605474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0605621Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0605625Z 
2025-09-07T09:16:27.0605714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0605897Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0605952Z     return mod(**inputs)
2025-09-07T09:16:27.0606181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0606246Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0606474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0606536Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0606749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0606816Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0607044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0607159Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0607387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0607455Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0607680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0607749Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0607752Z 
2025-09-07T09:16:27.0607839Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0608073Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0608131Z     return mod(**inputs)
2025-09-07T09:16:27.0608366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0608429Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0608656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0608725Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0608947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0609019Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0609281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0609351Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0609582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0609683Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0609912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0610012Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0610016Z 
2025-09-07T09:16:27.0610108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0610291Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0610348Z     return mod(**inputs)
2025-09-07T09:16:27.0610583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0610645Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0610876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0610939Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0611151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0611221Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0611447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0611523Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0611749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0611821Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0612049Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0612143Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0612148Z 
2025-09-07T09:16:27.0612241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0612423Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0612485Z     return mod(**inputs)
2025-09-07T09:16:27.0612754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0612817Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0613052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0613113Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0613332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0613399Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0613627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0613928Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0614159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0614236Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0614461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0614555Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0614559Z 
2025-09-07T09:16:27.0614647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0614828Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0614921Z     return mod(**inputs)
2025-09-07T09:16:27.0615153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0615219Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0615509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0615572Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0615792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0615858Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0616088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0616156Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0616387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0616459Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0616685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0616778Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0616781Z 
2025-09-07T09:16:27.0616854Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0616946Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0617138Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0617193Z     return mod(**inputs)
2025-09-07T09:16:27.0617426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0617486Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0617718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0617780Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0618037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0618110Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0618338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0618453Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0618679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0618776Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0619006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0619075Z     return self.weight * hidden_states
2025-09-07T09:16:27.0619078Z 
2025-09-07T09:16:27.0619170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0619354Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0619449Z     return mod(**inputs)
2025-09-07T09:16:27.0619686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0619748Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0619982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0620042Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0620257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0620322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0620582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0620657Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0620886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0620966Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0621224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0621300Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0621308Z 
2025-09-07T09:16:27.0621399Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0621582Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0621642Z     return mod(**inputs)
2025-09-07T09:16:27.0621871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0621936Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0622163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0622227Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0622443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0622508Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0622739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0622810Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0623034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0623106Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0623333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0623406Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0623409Z 
2025-09-07T09:16:27.0623498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0623682Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0623740Z     return mod(**inputs)
2025-09-07T09:16:27.0623968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0624067Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0624298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0624364Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0624577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0624642Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0624872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0624974Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0625207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0625279Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0625506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0625629Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0625632Z 
2025-09-07T09:16:27.0625722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0625909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0625965Z     return mod(**inputs)
2025-09-07T09:16:27.0626246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0626312Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0626546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0626642Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0626858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0626932Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0627159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0627228Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0627459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0627530Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0627761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0627905Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0627909Z 
2025-09-07T09:16:27.0628004Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0628187Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0628246Z     return mod(**inputs)
2025-09-07T09:16:27.0628477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0628538Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0628774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0628837Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0629050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0629117Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0629345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0629421Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0629644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0629749Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0629980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0630050Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0630053Z 
2025-09-07T09:16:27.0630143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0630325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0630384Z     return mod(**inputs)
2025-09-07T09:16:27.0630648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0630712Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0630946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0631007Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0631220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0631284Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0631508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0631582Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0631839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0631915Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0632143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0632270Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0632281Z 
2025-09-07T09:16:27.0632372Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0632555Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0632614Z     return mod(**inputs)
2025-09-07T09:16:27.0632842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0632908Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0633136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0633198Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0633414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0633481Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0633708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0633777Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0634001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0634076Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0634301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0634393Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0634397Z 
2025-09-07T09:16:27.0634484Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0634670Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0634728Z     return mod(**inputs)
2025-09-07T09:16:27.0634957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0635055Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0635285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0635347Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0635571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0635636Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0635868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0635941Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0636207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0636284Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0636510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0636612Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0636616Z 
2025-09-07T09:16:27.0636708Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0636895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0636954Z     return mod(**inputs)
2025-09-07T09:16:27.0637223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0637286Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0637519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0637587Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0637829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0637907Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0638136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0638206Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0638440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0638510Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0638741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0638807Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0638812Z 
2025-09-07T09:16:27.0638885Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0638979Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0639163Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0639228Z     return mod(**inputs)
2025-09-07T09:16:27.0639458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0639527Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0639757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0639817Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0640032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0640096Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0640328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0640411Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0640636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0640763Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0640991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0641063Z     return self.weight * hidden_states
2025-09-07T09:16:27.0641066Z 
2025-09-07T09:16:27.0641154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0641336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0641398Z     return mod(**inputs)
2025-09-07T09:16:27.0641629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0641731Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0641964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0642033Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0642245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0642312Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0642542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0642617Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0642882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0642992Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0643225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0643331Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0643335Z 
2025-09-07T09:16:27.0643422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0643612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0643671Z     return mod(**inputs)
2025-09-07T09:16:27.0643899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0643966Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0644200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0644263Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0644475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0644545Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0644770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0644850Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0645080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0645183Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0645433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0645504Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0645507Z 
2025-09-07T09:16:27.0645599Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0645786Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0645842Z     return mod(**inputs)
2025-09-07T09:16:27.0646076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0646142Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0646418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0646482Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0646696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0646768Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0646996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0647083Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0647307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0647463Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0647696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0647766Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0647770Z 
2025-09-07T09:16:27.0647844Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0647932Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0648118Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0648183Z     return mod(**inputs)
2025-09-07T09:16:27.0648442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0648514Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0648747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0648817Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0649061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0649130Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0649363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0649435Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0649665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0649762Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0649989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0650060Z     return self.weight * hidden_states
2025-09-07T09:16:27.0650065Z 
2025-09-07T09:16:27.0650153Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0650345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0650403Z     return mod(**inputs)
2025-09-07T09:16:27.0650634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0650704Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0650935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0651003Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0651219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0651291Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0651518Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0651592Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0651825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0651940Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0652173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0652240Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0652243Z 
2025-09-07T09:16:27.0652331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0652517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0652574Z     return mod(**inputs)
2025-09-07T09:16:27.0652810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0652906Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0653140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0653206Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0653421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0653492Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0653722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0653794Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0654057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0654133Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0654364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0654434Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0654437Z 
2025-09-07T09:16:27.0654561Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0654748Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0654807Z     return mod(**inputs)
2025-09-07T09:16:27.0655043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0655105Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0655341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0655404Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0655617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0655693Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0655921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0655994Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0656221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0656294Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0656520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0656640Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0656644Z 
2025-09-07T09:16:27.0656741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0656923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0656989Z     return mod(**inputs)
2025-09-07T09:16:27.0657217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0657282Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0657512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0657611Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0657832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0657898Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0658189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0658261Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0658486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0658598Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0658830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0658980Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0658985Z 
2025-09-07T09:16:27.0659078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0659262Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0659324Z     return mod(**inputs)
2025-09-07T09:16:27.0659553Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0659621Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0659881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0659946Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0660166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0660267Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0660502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0660570Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0660802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0660872Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0661102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0661176Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0661180Z 
2025-09-07T09:16:27.0661269Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0661456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0661515Z     return mod(**inputs)
2025-09-07T09:16:27.0661741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0661815Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0662042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0662110Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0662320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0662394Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0662624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0662694Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0662928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0662998Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0663262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0663359Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0663362Z 
2025-09-07T09:16:27.0663450Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0663641Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0663701Z     return mod(**inputs)
2025-09-07T09:16:27.0663936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0664000Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0664266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0664336Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0664549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0664623Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0664851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0664926Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0665153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0665222Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0665490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0665585Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0665588Z 
2025-09-07T09:16:27.0665680Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0665894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0665956Z     return mod(**inputs)
2025-09-07T09:16:27.0666190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0666252Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0666487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0666550Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0666762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0666837Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0667066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0667143Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0667367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0667440Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0667667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0667759Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0667763Z 
2025-09-07T09:16:27.0667856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0668042Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0668102Z     return mod(**inputs)
2025-09-07T09:16:27.0668330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0668395Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0668630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0668741Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0668958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0669025Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0669259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0669329Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0669561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0669636Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0669896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0669971Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0669975Z 
2025-09-07T09:16:27.0670063Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0670248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0670311Z     return mod(**inputs)
2025-09-07T09:16:27.0670539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0670607Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0670869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0670931Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0671152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0671221Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0671485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0671556Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0671788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.0671911Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0671914Z 
2025-09-07T09:16:27.0671983Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0672078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0672265Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0672328Z     return mod(**inputs)
2025-09-07T09:16:27.0672557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0672621Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0672857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0672918Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0673136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0673202Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0673428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0673504Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0673731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0673834Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0674062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0674135Z     return self.weight * hidden_states
2025-09-07T09:16:27.0674138Z 
2025-09-07T09:16:27.0674223Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0674456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0674515Z     return mod(**inputs)
2025-09-07T09:16:27.0674747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0674811Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0675045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0675109Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0675323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0675424Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0675662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0675734Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0675958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0676035Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0676261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0676335Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0676338Z 
2025-09-07T09:16:27.0676458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0676645Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0676705Z     return mod(**inputs)
2025-09-07T09:16:27.0676965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0677033Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0677265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0677332Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0677543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0677612Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0677845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0677914Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0678142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0678215Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0678441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0678515Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0678518Z 
2025-09-07T09:16:27.0678607Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0678793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0678851Z     return mod(**inputs)
2025-09-07T09:16:27.0679085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0679148Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0679381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0679452Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0679663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0679734Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0679997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0680069Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0680304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0680376Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0680607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0680724Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0680728Z 
2025-09-07T09:16:27.0680855Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0681041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0681101Z     return mod(**inputs)
2025-09-07T09:16:27.0681334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0681399Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0681632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0681693Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0681909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0682010Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0682240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0682315Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0682575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0682648Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0682877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0683023Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0683027Z 
2025-09-07T09:16:27.0683120Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0683301Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0683360Z     return mod(**inputs)
2025-09-07T09:16:27.0683593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0683655Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0683887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0683951Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0684166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0684231Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0684460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0684535Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0684765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0684843Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0685068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0685139Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0685149Z 
2025-09-07T09:16:27.0685242Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0685424Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0685521Z     return mod(**inputs)
2025-09-07T09:16:27.0685751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0685818Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0686046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0686107Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0686330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0686399Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0686667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0686737Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0686963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0687043Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0687266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0687368Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0687372Z 
2025-09-07T09:16:27.0687464Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0687679Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0687739Z     return mod(**inputs)
2025-09-07T09:16:27.0687970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0688041Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0688317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0688387Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0688604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0688673Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0688909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0688981Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0689213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0689287Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0689515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0689616Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0689620Z 
2025-09-07T09:16:27.0689714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0689899Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0689956Z     return mod(**inputs)
2025-09-07T09:16:27.0690193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0690257Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0690487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0690558Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0690770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0690843Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0691068Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0691174Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0691408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0691481Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0691714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0691805Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0691808Z 
2025-09-07T09:16:27.0691904Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0692122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0692181Z     return mod(**inputs)
2025-09-07T09:16:27.0692419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0692483Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0692721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0692785Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0692996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0693068Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0693328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0693409Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0693640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0693744Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0693981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0694051Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0694055Z 
2025-09-07T09:16:27.0694132Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0694220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0694411Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0694468Z     return mod(**inputs)
2025-09-07T09:16:27.0694703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0694770Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0695002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0695071Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0695284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0695351Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0695584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0695665Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0695900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0695995Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0696222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0696295Z     return self.weight * hidden_states
2025-09-07T09:16:27.0696298Z 
2025-09-07T09:16:27.0696387Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0696576Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0696671Z     return mod(**inputs)
2025-09-07T09:16:27.0696908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0696972Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0697200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0697271Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0697484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0697560Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0697823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0697907Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0698182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0698293Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0698528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0698600Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0698603Z 
2025-09-07T09:16:27.0698698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0698916Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0698973Z     return mod(**inputs)
2025-09-07T09:16:27.0699214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0699279Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0699551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0699614Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0699827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0699904Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0700132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0700213Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0700442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0700545Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0700779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0700853Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0700856Z 
2025-09-07T09:16:27.0700952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0701135Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0701199Z     return mod(**inputs)
2025-09-07T09:16:27.0701435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0701498Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0701737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0701798Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0702017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0702085Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0702316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0702431Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0702660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0702767Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0702995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0703195Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0703206Z 
2025-09-07T09:16:27.0703286Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0703382Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0703654Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0703712Z     return mod(**inputs)
2025-09-07T09:16:27.0703952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0704020Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0704249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0704321Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0704535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0704606Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0704914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0704989Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0705230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0705372Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0705616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0705688Z     return self.weight * hidden_states
2025-09-07T09:16:27.0705693Z 
2025-09-07T09:16:27.0705790Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0705988Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0706047Z     return mod(**inputs)
2025-09-07T09:16:27.0706288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0706354Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0706594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0706661Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0706876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0706955Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0707188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0707266Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0707490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0707566Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0707801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0707872Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0707877Z 
2025-09-07T09:16:27.0707977Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0708162Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0708228Z     return mod(**inputs)
2025-09-07T09:16:27.0708520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0708588Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0708825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0708889Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0709112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0709182Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0709411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0709520Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0709753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0709834Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0710059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0710128Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0710131Z 
2025-09-07T09:16:27.0710228Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0710411Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0710526Z     return mod(**inputs)
2025-09-07T09:16:27.0710761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0710829Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0711101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0711167Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0711392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0711463Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0711697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0711770Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0712002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0712083Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0712311Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0712435Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0712440Z 
2025-09-07T09:16:27.0712532Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0712719Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0712782Z     return mod(**inputs)
2025-09-07T09:16:27.0713010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0713083Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0713312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0713384Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0713594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0713664Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0713912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0713981Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0714256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0714329Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0714562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0714723Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0714726Z 
2025-09-07T09:16:27.0714832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0715022Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0715112Z     return mod(**inputs)
2025-09-07T09:16:27.0715349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0715420Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0715650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0715720Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0715932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0716006Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0716233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0716335Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0716573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0716651Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0716917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0716992Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0716995Z 
2025-09-07T09:16:27.0717089Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0717284Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0717345Z     return mod(**inputs)
2025-09-07T09:16:27.0717585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0717652Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0717887Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0717958Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0718174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0718253Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0718484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0718563Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0718791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0718864Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0719099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0719199Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0719202Z 
2025-09-07T09:16:27.0719301Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0719489Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0719549Z     return mod(**inputs)
2025-09-07T09:16:27.0719785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0719886Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0720123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0720188Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0720410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0720480Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0720710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0720819Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0721048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0721128Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0721356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0721451Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0721454Z 
2025-09-07T09:16:27.0721552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0721735Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0721800Z     return mod(**inputs)
2025-09-07T09:16:27.0722060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0722125Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0722367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0722463Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0722684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0722758Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0722992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0723066Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0723293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0723372Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0723602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0723709Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0723712Z 
2025-09-07T09:16:27.0723806Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0723993Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0724060Z     return mod(**inputs)
2025-09-07T09:16:27.0724292Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0724362Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0724593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0724658Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0724876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0724945Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0725180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0725254Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0725488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0725594Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0725822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0725898Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0725901Z 
2025-09-07T09:16:27.0725973Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0726072Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0726259Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0726318Z     return mod(**inputs)
2025-09-07T09:16:27.0726590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0726657Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0726891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0726956Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0727169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0727245Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0727470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0727548Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0727808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0727913Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0728173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0728244Z     return self.weight * hidden_states
2025-09-07T09:16:27.0728248Z 
2025-09-07T09:16:27.0728344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0728529Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0728593Z     return mod(**inputs)
2025-09-07T09:16:27.0728826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0728893Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0729132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0729193Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0729415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0729482Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0729717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0729794Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0730021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0730098Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0730325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0730398Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0730402Z 
2025-09-07T09:16:27.0730493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0730674Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0730737Z     return mod(**inputs)
2025-09-07T09:16:27.0730968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0731071Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0731305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0731367Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0731584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0731651Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0731886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0731957Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0732237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0732311Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0732539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0732612Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0732616Z 
2025-09-07T09:16:27.0732705Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0732894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0732951Z     return mod(**inputs)
2025-09-07T09:16:27.0733183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0733281Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0733512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0733582Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0733822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0733891Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0734124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0734195Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0734427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0734500Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0734734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0734852Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0734857Z 
2025-09-07T09:16:27.0734946Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0735133Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0735191Z     return mod(**inputs)
2025-09-07T09:16:27.0735425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0735487Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0735715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0735779Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0735989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0736059Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0736289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0736358Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0736591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0736695Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0736923Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0737068Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0737071Z 
2025-09-07T09:16:27.0737167Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0737351Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0737412Z     return mod(**inputs)
2025-09-07T09:16:27.0737669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0737765Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0738062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0738126Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0738339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0738409Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0738639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0738715Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0738978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0739058Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0739286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0739357Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0739360Z 
2025-09-07T09:16:27.0739488Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0739677Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0739741Z     return mod(**inputs)
2025-09-07T09:16:27.0739973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0740036Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0740270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0740333Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0740552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0740619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0740850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0740924Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0741152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0741229Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0741458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0741562Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0741565Z 
2025-09-07T09:16:27.0741658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0741843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0741905Z     return mod(**inputs)
2025-09-07T09:16:27.0742132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0742201Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0742434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0742528Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0742746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0742813Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0743040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0743110Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0743338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0743452Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0743682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0743781Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0743786Z 
2025-09-07T09:16:27.0743877Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0744067Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0744123Z     return mod(**inputs)
2025-09-07T09:16:27.0744354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0744421Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0744684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0744754Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0744968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0745070Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0745306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0745380Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0745615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0745686Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0745918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0746015Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0746018Z 
2025-09-07T09:16:27.0746109Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0746298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0746354Z     return mod(**inputs)
2025-09-07T09:16:27.0746592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0746657Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0746885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0746949Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0747158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0747231Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0747458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0747531Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0747761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0747833Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0748061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0748162Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0748165Z 
2025-09-07T09:16:27.0748256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0748437Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0748495Z     return mod(**inputs)
2025-09-07T09:16:27.0748730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0748791Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0749025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0749118Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0749333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0749403Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0749630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0749703Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0749927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:16:27.0750053Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0750094Z 
2025-09-07T09:16:27.0750166Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0750256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0750441Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0750528Z     return mod(**inputs)
2025-09-07T09:16:27.0750766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0750833Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0751063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0751129Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0751340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0751408Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0751638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0751721Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0751952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0752040Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0752270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0752335Z     return self.weight * hidden_states
2025-09-07T09:16:27.0752338Z 
2025-09-07T09:16:27.0752433Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0752614Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0752673Z     return mod(**inputs)
2025-09-07T09:16:27.0752909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0752970Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0753207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0753266Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0753481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0753600Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0753827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0753913Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0754138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0754246Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0754479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0754582Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0754585Z 
2025-09-07T09:16:27.0754678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0754859Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0754920Z     return mod(**inputs)
2025-09-07T09:16:27.0755150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0755213Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0762433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0762536Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0762855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0762943Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0763201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0763480Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0763736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0763851Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0764089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0764166Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0764170Z 
2025-09-07T09:16:27.0764274Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0764481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0764544Z     return mod(**inputs)
2025-09-07T09:16:27.0764791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0764865Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0765101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0765173Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0765392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0765468Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0765699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0765786Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0766017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0766125Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0766361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0766440Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0766443Z 
2025-09-07T09:16:27.0766566Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0766665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0766862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0766927Z     return mod(**inputs)
2025-09-07T09:16:27.0767168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0767242Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0767474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0767538Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0767801Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0767876Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0768105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0768178Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0768405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0768503Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0768728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0768829Z     return self.weight * hidden_states
2025-09-07T09:16:27.0768833Z 
2025-09-07T09:16:27.0768928Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0769120Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0769177Z     return mod(**inputs)
2025-09-07T09:16:27.0769439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0769507Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0769734Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0769799Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0770011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0770081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0770310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0770381Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0770613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0770689Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0770917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0770986Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0770990Z 
2025-09-07T09:16:27.0771079Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0771264Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0771320Z     return mod(**inputs)
2025-09-07T09:16:27.0771552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0771616Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0771845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0771909Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0772120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0772223Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0772449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0772520Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0772744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0772813Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0773051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0773124Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0773159Z 
2025-09-07T09:16:27.0773256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0773450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0773507Z     return mod(**inputs)
2025-09-07T09:16:27.0773750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0773814Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0774046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0774106Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0774351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0774423Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0774654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0774729Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0774988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0775065Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0775291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0775414Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0775417Z 
2025-09-07T09:16:27.0775509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0775695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0775755Z     return mod(**inputs)
2025-09-07T09:16:27.0775984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0776050Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0776281Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0776341Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0776557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0776621Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0776847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0776917Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0777145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0777217Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0777442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0777595Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0777598Z 
2025-09-07T09:16:27.0777689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0777910Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0778057Z     return mod(**inputs)
2025-09-07T09:16:27.0778296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0778364Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0778596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0778663Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0778883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0778988Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0779227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0779299Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0779537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0779610Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0779837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0779913Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0779917Z 
2025-09-07T09:16:27.0780045Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0780238Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0780297Z     return mod(**inputs)
2025-09-07T09:16:27.0780528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0780641Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0780875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0780947Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0781159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0781228Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0781462Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0781530Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0781766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0781839Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0782070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0782171Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0782176Z 
2025-09-07T09:16:27.0782266Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0782456Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0782513Z     return mod(**inputs)
2025-09-07T09:16:27.0782749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0782812Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0783047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0783111Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0783324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0783398Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0783625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0783735Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0783964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0784034Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0784267Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0784364Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0784367Z 
2025-09-07T09:16:27.0784466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0784685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0784746Z     return mod(**inputs)
2025-09-07T09:16:27.0784981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0785048Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0785284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0785348Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0785563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0785632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0785892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0785971Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0786232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0786306Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0786535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0786629Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0786632Z 
2025-09-07T09:16:27.0786723Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0786905Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0786962Z     return mod(**inputs)
2025-09-07T09:16:27.0787193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0787255Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0787490Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0787551Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0787765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0787832Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0788063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0788132Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0788360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0788436Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0788660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0788732Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0788735Z 
2025-09-07T09:16:27.0788806Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0788895Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0789079Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0789169Z     return mod(**inputs)
2025-09-07T09:16:27.0789403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0789464Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0789694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0789758Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0789969Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0790037Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0790298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0790373Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0790600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0790700Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0790929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0790994Z     return self.weight * hidden_states
2025-09-07T09:16:27.0790997Z 
2025-09-07T09:16:27.0791090Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0791310Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0791368Z     return mod(**inputs)
2025-09-07T09:16:27.0791605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0791700Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0791937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0791999Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0792214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0792281Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0792507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0792580Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0792806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0792884Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0793111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0793178Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0793183Z 
2025-09-07T09:16:27.0793281Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0793473Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0793532Z     return mod(**inputs)
2025-09-07T09:16:27.0793773Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0793835Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0794073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0794134Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0794347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0794417Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0794652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0794755Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0794981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0795056Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0795282Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0795352Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0795358Z 
2025-09-07T09:16:27.0795446Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0795630Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0795723Z     return mod(**inputs)
2025-09-07T09:16:27.0795955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0796024Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0796254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0796313Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0796528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0796596Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0796858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0796928Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0797160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0797236Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0797492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0797613Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0797617Z 
2025-09-07T09:16:27.0797706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0797902Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0797961Z     return mod(**inputs)
2025-09-07T09:16:27.0798201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0798272Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0798503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0798573Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0798790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0798864Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0799099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0799171Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0799398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0799474Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0799705Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0799851Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0799856Z 
2025-09-07T09:16:27.0799952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0800146Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0800204Z     return mod(**inputs)
2025-09-07T09:16:27.0800480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0800544Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0800776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0800846Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0801065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0801136Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0801363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0801527Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0801759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0801834Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0802062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0802134Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0802137Z 
2025-09-07T09:16:27.0802234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0802418Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0802522Z     return mod(**inputs)
2025-09-07T09:16:27.0802759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0802826Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0803291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0803355Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0803573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0803639Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0803867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0803935Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0804163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0804234Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0804458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0804561Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0804566Z 
2025-09-07T09:16:27.0804658Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0804849Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0804905Z     return mod(**inputs)
2025-09-07T09:16:27.0805138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0805203Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0805431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0805497Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0805707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0805775Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0806006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0806074Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0806357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0806430Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0806656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0806752Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0806756Z 
2025-09-07T09:16:27.0806851Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0807049Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0807156Z     return mod(**inputs)
2025-09-07T09:16:27.0807391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0807457Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0807684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0807750Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0807961Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0808028Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0808255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0808367Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0808598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0808670Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0808932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0809025Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0809030Z 
2025-09-07T09:16:27.0809122Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0809305Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0809362Z     return mod(**inputs)
2025-09-07T09:16:27.0809592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0809653Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0809885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0809948Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0810159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0810226Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0810450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0810522Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0810746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0810819Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0811043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0811113Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0811116Z 
2025-09-07T09:16:27.0811189Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0811279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0811466Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0811522Z     return mod(**inputs)
2025-09-07T09:16:27.0811751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0811853Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0812082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0812144Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0812353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0812421Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0812648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0812763Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0812993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0813078Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0813307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0813374Z     return self.weight * hidden_states
2025-09-07T09:16:27.0813378Z 
2025-09-07T09:16:27.0813466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0813650Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0813704Z     return mod(**inputs)
2025-09-07T09:16:27.0813967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0814030Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0814261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0814353Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0814563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0814632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0814861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0814936Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0815162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0815273Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0815501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0815572Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0815575Z 
2025-09-07T09:16:27.0815665Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0815845Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0815901Z     return mod(**inputs)
2025-09-07T09:16:27.0816131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0816192Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0816421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0816480Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0816691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0816758Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0816986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0817062Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0817326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0817428Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0817654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0817725Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0817728Z 
2025-09-07T09:16:27.0817820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0818053Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0818112Z     return mod(**inputs)
2025-09-07T09:16:27.0818379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0818444Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0818675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0818737Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0818951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0819017Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0819242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0819352Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0819580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0819684Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0819943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0820015Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0820020Z 
2025-09-07T09:16:27.0820108Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0820289Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0820346Z     return mod(**inputs)
2025-09-07T09:16:27.0820574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0820636Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0820865Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0820925Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0821138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0821205Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0821435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0821510Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0821735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:16:27.0821856Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:16:27.0821859Z 
2025-09-07T09:16:27.0821928Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0822019Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0822198Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0822257Z     return mod(**inputs)
2025-09-07T09:16:27.0822485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0822548Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0822778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0822871Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0823085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0823150Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0823375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0823452Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0823680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0823828Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0824055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0824123Z     return self.weight * hidden_states
2025-09-07T09:16:27.0824130Z 
2025-09-07T09:16:27.0824221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0824405Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0824465Z     return mod(**inputs)
2025-09-07T09:16:27.0824696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0824761Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0825023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0825086Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0825305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0825404Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0825639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0825712Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0825937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0826014Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0826240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0826312Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0826315Z 
2025-09-07T09:16:27.0826404Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0826593Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0826650Z     return mod(**inputs)
2025-09-07T09:16:27.0826883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0826948Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0827177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0827240Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0827463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0827528Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0827761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0827829Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0828062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0828140Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0828388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0828494Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0828497Z 
2025-09-07T09:16:27.0828585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0828773Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0828831Z     return mod(**inputs)
2025-09-07T09:16:27.0829067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0829131Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0829361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0829462Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0829677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0829749Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0829976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0830047Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0830274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0830342Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0830602Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0830724Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0830728Z 
2025-09-07T09:16:27.0830820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0831036Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0831094Z     return mod(**inputs)
2025-09-07T09:16:27.0831329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0831390Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0831622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0831684Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0831896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0831967Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0832191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0832266Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0832492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0832565Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0832793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0832940Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0832944Z 
2025-09-07T09:16:27.0833037Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0833222Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0833281Z     return mod(**inputs)
2025-09-07T09:16:27.0833512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0833575Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0833807Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0833901Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0834118Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0834183Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0834407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0834480Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0834708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0834780Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0835043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0835113Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0835118Z 
2025-09-07T09:16:27.0835211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0835400Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0835459Z     return mod(**inputs)
2025-09-07T09:16:27.0835688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0835756Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0835985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0836191Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0836412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0836480Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0836743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0836812Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0837043Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0837115Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0837340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0837441Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0837444Z 
2025-09-07T09:16:27.0837534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0837717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0837778Z     return mod(**inputs)
2025-09-07T09:16:27.0838009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0838076Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0838310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0838375Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0838586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0838652Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0838882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0838952Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0839180Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0839255Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0839481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0839611Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0839613Z 
2025-09-07T09:16:27.0839702Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0839886Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0839943Z     return mod(**inputs)
2025-09-07T09:16:27.0840179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0840243Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0840473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0840574Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0840784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0840855Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0841082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0841151Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0841381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0841449Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0841678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0841802Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0841806Z 
2025-09-07T09:16:27.0841894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0842081Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0842171Z     return mod(**inputs)
2025-09-07T09:16:27.0842406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0842470Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0842700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0842760Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0842972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0843040Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0843268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0843340Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0843572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0843643Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0843875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0843942Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0843945Z 
2025-09-07T09:16:27.0844023Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0844110Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0844294Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0844356Z     return mod(**inputs)
2025-09-07T09:16:27.0844586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0844651Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0844883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0844949Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0845162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0845278Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0845507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0845577Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0845805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0845900Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0846126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0846230Z     return self.weight * hidden_states
2025-09-07T09:16:27.0846233Z 
2025-09-07T09:16:27.0846322Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0846506Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0846566Z     return mod(**inputs)
2025-09-07T09:16:27.0846793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0846859Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0847093Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0847157Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0847396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0847470Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0847699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0847802Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0848037Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0848112Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0848340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0848407Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0848411Z 
2025-09-07T09:16:27.0848502Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0848690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0848747Z     return mod(**inputs)
2025-09-07T09:16:27.0848979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0849041Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0849274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0849340Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0849550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0849619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0849846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0849915Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0850140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0850212Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0850437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0850505Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0850508Z 
2025-09-07T09:16:27.0850632Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0850813Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0850869Z     return mod(**inputs)
2025-09-07T09:16:27.0851098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0851159Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0851394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0851454Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0851664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0851765Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0851993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0852066Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0852290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0852362Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0852585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0852700Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0852736Z 
2025-09-07T09:16:27.0852828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0853010Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0853070Z     return mod(**inputs)
2025-09-07T09:16:27.0853329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0853394Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0853626Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0853687Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0853900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0853964Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0854193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0854262Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0854488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0854562Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0854786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0854930Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0854934Z 
2025-09-07T09:16:27.0855020Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0855201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0855257Z     return mod(**inputs)
2025-09-07T09:16:27.0855486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0855552Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0855785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0855848Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0856065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0856167Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0856400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0856469Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0856698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0856771Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0856996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0857069Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0857104Z 
2025-09-07T09:16:27.0857192Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0857380Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0857438Z     return mod(**inputs)
2025-09-07T09:16:27.0857670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0857735Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0857963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0858095Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0858347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0858417Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0858651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0858724Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0858988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0859063Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0859296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0859390Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0859393Z 
2025-09-07T09:16:27.0859485Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0859671Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0859731Z     return mod(**inputs)
2025-09-07T09:16:27.0859964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0860028Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0860259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0860323Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0860535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0860604Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0860829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0860902Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0861133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0861206Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0861446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0861538Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0861542Z 
2025-09-07T09:16:27.0861635Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0861852Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0861910Z     return mod(**inputs)
2025-09-07T09:16:27.0862146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0862209Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0862442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0862508Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0862720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0862827Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0863056Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0863130Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0863356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0863431Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0863656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0863752Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0863755Z 
2025-09-07T09:16:27.0863880Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0864068Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0864129Z     return mod(**inputs)
2025-09-07T09:16:27.0864360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0864467Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0864707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0864770Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0864989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0865056Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0865280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0865356Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0865583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0865660Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0865888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0865957Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0865962Z 
2025-09-07T09:16:27.0866030Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0866121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0866308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0866362Z     return mod(**inputs)
2025-09-07T09:16:27.0866595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0866660Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0866889Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0866955Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0867167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0867236Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0867506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0867587Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0867819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0867905Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0868134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0868201Z     return self.weight * hidden_states
2025-09-07T09:16:27.0868205Z 
2025-09-07T09:16:27.0868329Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0868512Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0868569Z     return mod(**inputs)
2025-09-07T09:16:27.0868805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0868871Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0869109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0869170Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0869382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0869493Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0869727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0869814Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0870075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0870196Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0870426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0870498Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0870502Z 
2025-09-07T09:16:27.0870594Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0870778Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0870839Z     return mod(**inputs)
2025-09-07T09:16:27.0871072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0871138Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0871375Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0871438Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0871654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0871722Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0871950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0872029Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0872259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0872372Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0872600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0872683Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0872686Z 
2025-09-07T09:16:27.0872774Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0872958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0873054Z     return mod(**inputs)
2025-09-07T09:16:27.0873286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0873353Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0873581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0873643Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0873860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0873957Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0874188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0874263Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0874494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0874600Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0874825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0874899Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0874903Z 
2025-09-07T09:16:27.0874974Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0875098Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0875282Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0875341Z     return mod(**inputs)
2025-09-07T09:16:27.0875609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0875673Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0875911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0875976Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0876187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0876260Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0876488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0876567Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0876793Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0876894Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0877120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0877189Z     return self.weight * hidden_states
2025-09-07T09:16:27.0877192Z 
2025-09-07T09:16:27.0877286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0877467Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0877530Z     return mod(**inputs)
2025-09-07T09:16:27.0877762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0877825Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0878061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0878122Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0878340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0878404Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0878680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0878755Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0878980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0879057Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0879284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0879356Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0879359Z 
2025-09-07T09:16:27.0879449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0879666Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0879730Z     return mod(**inputs)
2025-09-07T09:16:27.0879959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0880029Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0880258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0880318Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0880535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0880601Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0880870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0880944Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0881205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0881287Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0881516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0881590Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0881595Z 
2025-09-07T09:16:27.0881681Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0881869Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0881927Z     return mod(**inputs)
2025-09-07T09:16:27.0882159Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0882231Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0882464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0882532Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0882746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0882818Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0883051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0883122Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0883350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0883423Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0883650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0883777Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0883780Z 
2025-09-07T09:16:27.0883870Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0884063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0884156Z     return mod(**inputs)
2025-09-07T09:16:27.0884397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0884460Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0884692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0884757Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0884971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0885043Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0885305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0885380Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0885612Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0885686Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0885921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0886066Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0886070Z 
2025-09-07T09:16:27.0886168Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0886410Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0886469Z     return mod(**inputs)
2025-09-07T09:16:27.0886719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0886784Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0887054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0887119Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0887333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0887407Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0887638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0887715Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0887944Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0888015Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0888246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0888317Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0888321Z 
2025-09-07T09:16:27.0888422Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0888608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0888670Z     return mod(**inputs)
2025-09-07T09:16:27.0888899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0888961Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0889198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0889261Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0889474Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0889544Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0889771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0889909Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0890137Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0890215Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0890441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0890538Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0890549Z 
2025-09-07T09:16:27.0890641Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0890827Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0890928Z     return mod(**inputs)
2025-09-07T09:16:27.0891162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0891231Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0891464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0891525Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0891739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0891806Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0892071Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0892144Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0892372Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0892448Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0892706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0892806Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0892809Z 
2025-09-07T09:16:27.0892896Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0893081Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0893138Z     return mod(**inputs)
2025-09-07T09:16:27.0893368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0893438Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0893666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0893734Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0893946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0894013Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0894246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0894316Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0894542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0894611Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0894839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0894938Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0894942Z 
2025-09-07T09:16:27.0895032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0895221Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0895279Z     return mod(**inputs)
2025-09-07T09:16:27.0895554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0895618Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0895850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0895921Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0896133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0896211Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0896435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0896542Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0896778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0896850Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0897085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0897153Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0897157Z 
2025-09-07T09:16:27.0897256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0897440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0897497Z     return mod(**inputs)
2025-09-07T09:16:27.0897769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0897835Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0898136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0898231Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0898446Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0898523Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0898751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0898826Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0899054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.0899182Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0899185Z 
2025-09-07T09:16:27.0899264Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0899356Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0899543Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0899600Z     return mod(**inputs)
2025-09-07T09:16:27.0899829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0899898Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0900128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0900195Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0900408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0900480Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0900704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0900777Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0901011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0901107Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0901381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0901449Z     return self.weight * hidden_states
2025-09-07T09:16:27.0901452Z 
2025-09-07T09:16:27.0901546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0901736Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0901795Z     return mod(**inputs)
2025-09-07T09:16:27.0902032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0902131Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0902370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0902434Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0902646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0902719Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0902948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0903023Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0903400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0903529Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0903771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0903843Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0903847Z 
2025-09-07T09:16:27.0903991Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0904183Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0904247Z     return mod(**inputs)
2025-09-07T09:16:27.0904491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0904558Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0904794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0904862Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0905085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0905154Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0905385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0905462Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0905689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0905765Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0905991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0906061Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0906064Z 
2025-09-07T09:16:27.0906162Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0906350Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0906413Z     return mod(**inputs)
2025-09-07T09:16:27.0906644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0906710Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0906942Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0907055Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0907276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0907343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0907581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0907651Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0907880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0907956Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0908248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0908380Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0908385Z 
2025-09-07T09:16:27.0908474Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0908662Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0908723Z     return mod(**inputs)
2025-09-07T09:16:27.0908953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0909021Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0909291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0909358Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0909579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0909644Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0909910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0909981Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0910210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0910281Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0910507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0910661Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0910664Z 
2025-09-07T09:16:27.0910755Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0910945Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0911000Z     return mod(**inputs)
2025-09-07T09:16:27.0911231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0911301Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0911533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0911602Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0911816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0911887Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0912119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0912192Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0912428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0912509Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0912739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0912844Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0912847Z 
2025-09-07T09:16:27.0912937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0913125Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0913181Z     return mod(**inputs)
2025-09-07T09:16:27.0913417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0913479Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0913712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0913808Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0914024Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0914099Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0914325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0914396Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0914622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0914694Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0914958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0915061Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0915066Z 
2025-09-07T09:16:27.0915180Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0915413Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0915479Z     return mod(**inputs)
2025-09-07T09:16:27.0915717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0915780Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0916026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0916091Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0916310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0916385Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0916617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0916694Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0916925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0917002Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0917227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0917318Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0917321Z 
2025-09-07T09:16:27.0917413Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0917602Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0917671Z     return mod(**inputs)
2025-09-07T09:16:27.0917906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0917968Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0918203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0918267Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0918528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0918597Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0918826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0918897Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0919125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0919199Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0919425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0919558Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0919562Z 
2025-09-07T09:16:27.0919651Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0919836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0919898Z     return mod(**inputs)
2025-09-07T09:16:27.0920129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0920193Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0920419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0920512Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0920729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0920797Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0921060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0921133Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0921371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0921441Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0921667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0921736Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0921740Z 
2025-09-07T09:16:27.0921813Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0921906Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0922091Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0922148Z     return mod(**inputs)
2025-09-07T09:16:27.0922384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0922448Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0922681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0922743Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0922956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0923026Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0923252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0923342Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0923568Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0923663Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0923891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0923999Z     return self.weight * hidden_states
2025-09-07T09:16:27.0924002Z 
2025-09-07T09:16:27.0924101Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0924287Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0924349Z     return mod(**inputs)
2025-09-07T09:16:27.0924577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0924641Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0924875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0924971Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0925191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0925258Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0925489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0925576Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0925805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0925924Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0926181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0926263Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0926268Z 
2025-09-07T09:16:27.0926359Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0926574Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0926636Z     return mod(**inputs)
2025-09-07T09:16:27.0926868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0926938Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0927174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0927236Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0927454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0927524Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0927755Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0927835Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0928069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0928174Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0928402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0928480Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0928484Z 
2025-09-07T09:16:27.0928573Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0928760Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0928819Z     return mod(**inputs)
2025-09-07T09:16:27.0929050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0929117Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0929343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0929409Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0929679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0929748Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0929985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0930066Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0930301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0930404Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0930634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0930735Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0930738Z 
2025-09-07T09:16:27.0930812Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0930906Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0931094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0931153Z     return mod(**inputs)
2025-09-07T09:16:27.0931386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0931448Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0931720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0931784Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0932000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0932068Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0932329Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0932409Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0932636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0932736Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0932962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0933035Z     return self.weight * hidden_states
2025-09-07T09:16:27.0933038Z 
2025-09-07T09:16:27.0933133Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0933315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0933378Z     return mod(**inputs)
2025-09-07T09:16:27.0933609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0933676Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0933904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0933969Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0934183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0934251Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0934485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0934556Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0934787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0934863Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0935088Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0935197Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0935201Z 
2025-09-07T09:16:27.0935291Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0935476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0935532Z     return mod(**inputs)
2025-09-07T09:16:27.0935767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0935836Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0936063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0936166Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0936379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0936447Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0936680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0936751Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0936980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0937050Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0937318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0937387Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0937391Z 
2025-09-07T09:16:27.0937477Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0937666Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0937756Z     return mod(**inputs)
2025-09-07T09:16:27.0938050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0938117Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0938349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0938418Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0938630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0938704Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0938935Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0939011Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0939252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0939326Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0939567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0939686Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0939689Z 
2025-09-07T09:16:27.0939786Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0939972Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0940033Z     return mod(**inputs)
2025-09-07T09:16:27.0940275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0940340Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0940573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0940636Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0940854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0940984Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0941215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0941291Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0941520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0941595Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0941827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0942009Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0942012Z 
2025-09-07T09:16:27.0942113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0942299Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0942363Z     return mod(**inputs)
2025-09-07T09:16:27.0942592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0942659Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0942895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0942957Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0943212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0943282Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0943512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0943621Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0943852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0943930Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0944160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0944236Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0944239Z 
2025-09-07T09:16:27.0944331Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0944517Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0944575Z     return mod(**inputs)
2025-09-07T09:16:27.0944805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0944873Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0945104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0945168Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0945380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0945448Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0945680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0945749Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0945975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0946051Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0946277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0946383Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0946387Z 
2025-09-07T09:16:27.0946511Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0946701Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0946758Z     return mod(**inputs)
2025-09-07T09:16:27.0946987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0947051Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0947283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0947348Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0947558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0947660Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0947900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0947970Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0948200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0948267Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0948492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0948590Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0948625Z 
2025-09-07T09:16:27.0948718Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0948907Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0948966Z     return mod(**inputs)
2025-09-07T09:16:27.0949246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0949311Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0949546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0949614Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0949827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0949900Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0950127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0950196Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0950428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0950500Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0950733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0950827Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0950830Z 
2025-09-07T09:16:27.0950924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0951106Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0951167Z     return mod(**inputs)
2025-09-07T09:16:27.0951403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0951466Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0951703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0951767Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0951980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0952054Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0952323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0952400Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0952628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0952700Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0952932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0953002Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0953006Z 
2025-09-07T09:16:27.0953119Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0953212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0953405Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0953466Z     return mod(**inputs)
2025-09-07T09:16:27.0953698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0953766Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0953997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0954069Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0954318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0954390Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0954624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0954697Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0954962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.0955064Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0955288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0955363Z     return self.weight * hidden_states
2025-09-07T09:16:27.0955366Z 
2025-09-07T09:16:27.0955458Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0955647Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0955704Z     return mod(**inputs)
2025-09-07T09:16:27.0955943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0956009Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0956242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0956312Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0956528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0956601Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0956830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0956899Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0957134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0957209Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0957437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0957506Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0957511Z 
2025-09-07T09:16:27.0957605Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0957823Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0957881Z     return mod(**inputs)
2025-09-07T09:16:27.0958111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0958172Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0958403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0958470Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0958682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0958786Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0959015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0959091Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0959319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0959394Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0959624Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.0959690Z     key_states = self.k(current_states)
2025-09-07T09:16:27.0959694Z 
2025-09-07T09:16:27.0959783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0959995Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0960060Z     return mod(**inputs)
2025-09-07T09:16:27.0960291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0960389Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0960629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0960691Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0960906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0960971Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0961196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0961269Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0961496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0961571Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0961797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.0961913Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.0961924Z 
2025-09-07T09:16:27.0962011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0962191Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0962250Z     return mod(**inputs)
2025-09-07T09:16:27.0962495Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0962564Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0962796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0962860Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0963076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0963144Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0963374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0963482Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0963711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0963787Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0964021Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.0964179Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.0964182Z 
2025-09-07T09:16:27.0964273Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0964497Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0964554Z     return mod(**inputs)
2025-09-07T09:16:27.0964784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0964853Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0965083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0965149Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0965362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0965428Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0965694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0965769Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0966031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0966108Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0966334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.0966409Z     value_states = self.v(current_states)
2025-09-07T09:16:27.0966413Z 
2025-09-07T09:16:27.0966502Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0966686Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0966742Z     return mod(**inputs)
2025-09-07T09:16:27.0966978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0967041Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0967273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0967335Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0967549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0967616Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0967842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0967910Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0968136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0968206Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0968433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0968530Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0968533Z 
2025-09-07T09:16:27.0968623Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0968808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0968903Z     return mod(**inputs)
2025-09-07T09:16:27.0969135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0969196Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0969428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0969488Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0969700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0969767Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0970029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0970104Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0970332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0970405Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0970634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.0970725Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.0970729Z 
2025-09-07T09:16:27.0970818Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0971044Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0971100Z     return mod(**inputs)
2025-09-07T09:16:27.0971332Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0971396Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0971657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0971722Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0971939Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0972005Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0972231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0972303Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0972529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0972604Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0972829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.0972925Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.0972928Z 
2025-09-07T09:16:27.0973024Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0973207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0973271Z     return mod(**inputs)
2025-09-07T09:16:27.0973502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0973569Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0973804Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0973865Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0974080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0974150Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0974383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0974489Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0974717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.0974801Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.0975025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.0975095Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.0975099Z 
2025-09-07T09:16:27.0975194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0975377Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0982555Z     return mod(**inputs)
2025-09-07T09:16:27.0982872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0982947Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0983217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0983285Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0983510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0983591Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0983910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.0983989Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.0984220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:16:27.0984361Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.0984427Z 
2025-09-07T09:16:27.0984524Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0984634Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0984843Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0984906Z     return mod(**inputs)
2025-09-07T09:16:27.0985153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0985221Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0985456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0985529Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0985749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0985831Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0986066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0986151Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0986386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.0986477Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0986709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0986778Z     return self.weight * hidden_states
2025-09-07T09:16:27.0986782Z 
2025-09-07T09:16:27.0986884Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0987080Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0987143Z     return mod(**inputs)
2025-09-07T09:16:27.0987383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0987452Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0987838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0987903Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0988124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0988203Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0988440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0988528Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0988759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0988909Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0989144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.0989218Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.0989221Z 
2025-09-07T09:16:27.0989328Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0989526Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0989589Z     return mod(**inputs)
2025-09-07T09:16:27.0989822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0989920Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0990158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0990222Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0990479Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0990549Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0990781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0990867Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0991094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0991207Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0991438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.0991518Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.0991523Z 
2025-09-07T09:16:27.0991618Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0991807Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0991871Z     return mod(**inputs)
2025-09-07T09:16:27.0992104Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0992173Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0992401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0992464Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0992682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0992752Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0992989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.0993067Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.0993296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.0993436Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.0993662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.0993733Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.0993736Z 
2025-09-07T09:16:27.0993812Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.0993907Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0994094Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0994153Z     return mod(**inputs)
2025-09-07T09:16:27.0994390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0994489Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0994726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0994792Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0995006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0995077Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0995312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0995392Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0995654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.0995753Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.0995986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.0996087Z     return self.weight * hidden_states
2025-09-07T09:16:27.0996091Z 
2025-09-07T09:16:27.0996188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0996372Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0996437Z     return mod(**inputs)
2025-09-07T09:16:27.0996667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0996731Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0996968Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0997030Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0997248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0997314Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0997544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0997621Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.0997851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.0997931Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.0998156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.0998227Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.0998233Z 
2025-09-07T09:16:27.0998326Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.0998509Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.0998574Z     return mod(**inputs)
2025-09-07T09:16:27.0998805Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.0998873Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.0999136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.0999199Z     layer_outputs = layer_module(
2025-09-07T09:16:27.0999418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.0999484Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.0999715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.0999787Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1000012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1000138Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1000366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1000439Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1000442Z 
2025-09-07T09:16:27.1000530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1000719Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1000775Z     return mod(**inputs)
2025-09-07T09:16:27.1001002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1001073Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1001333Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1001398Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1001613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1001709Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1001948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1002019Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1002271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1002346Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1002577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1002700Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1002703Z 
2025-09-07T09:16:27.1002792Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1002982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1003221Z     return mod(**inputs)
2025-09-07T09:16:27.1003465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1003531Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1003763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1003831Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1004047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1004119Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1004350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1004426Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1004659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1004731Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1005042Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1005192Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1005196Z 
2025-09-07T09:16:27.1005296Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1005486Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1005545Z     return mod(**inputs)
2025-09-07T09:16:27.1005788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1005855Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1006143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1006208Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1006424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1006502Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1006731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1006806Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1007032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1007157Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1007387Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1007458Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1007462Z 
2025-09-07T09:16:27.1007598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1007790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1007850Z     return mod(**inputs)
2025-09-07T09:16:27.1008083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1008146Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1008379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1008441Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1008656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1008726Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1008952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1009025Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1009249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1009322Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1009556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1009661Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1009664Z 
2025-09-07T09:16:27.1009760Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1009958Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1010017Z     return mod(**inputs)
2025-09-07T09:16:27.1010257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1010326Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1010560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1010659Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1010880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1010951Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1011182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1011255Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1011484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1011561Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1011826Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1011928Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1011931Z 
2025-09-07T09:16:27.1012024Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1012213Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1012270Z     return mod(**inputs)
2025-09-07T09:16:27.1012501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1012566Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1012828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1012893Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1013108Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1013178Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1013438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1013511Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1013742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1013812Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1014041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1014135Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1014140Z 
2025-09-07T09:16:27.1014230Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1014418Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1014476Z     return mod(**inputs)
2025-09-07T09:16:27.1014707Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1014769Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1014999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1015062Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1015275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1015343Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1015570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1015639Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1015867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1015938Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1016165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1016267Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1016270Z 
2025-09-07T09:16:27.1016344Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1016432Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1016616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1016674Z     return mod(**inputs)
2025-09-07T09:16:27.1016904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1016968Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1017195Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1017290Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1017506Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1017575Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1017803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1017872Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1018139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1018237Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1018501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1018572Z     return self.weight * hidden_states
2025-09-07T09:16:27.1018576Z 
2025-09-07T09:16:27.1018664Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1018880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1018941Z     return mod(**inputs)
2025-09-07T09:16:27.1019172Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1019237Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1019467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1019531Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1019745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1019811Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1020039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1020109Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1020341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1020414Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1020637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1020708Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1020711Z 
2025-09-07T09:16:27.1020799Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1020983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1021039Z     return mod(**inputs)
2025-09-07T09:16:27.1021270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1021333Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1021565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1021627Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1021895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1021963Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1022188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1022258Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1022489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1022559Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1022785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1022885Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1022889Z 
2025-09-07T09:16:27.1022981Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1023166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1023221Z     return mod(**inputs)
2025-09-07T09:16:27.1023455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1023519Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1023751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1023847Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1024063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1024129Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1024388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1024457Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1024687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1024761Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1024989Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1025114Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1025117Z 
2025-09-07T09:16:27.1025207Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1025392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1025449Z     return mod(**inputs)
2025-09-07T09:16:27.1025680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1025747Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1025976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1026039Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1026250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1026315Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1026543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1026612Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1026842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1026914Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1027140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1027288Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1027328Z 
2025-09-07T09:16:27.1027416Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1027601Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1027656Z     return mod(**inputs)
2025-09-07T09:16:27.1027891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1027952Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1028182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1028277Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1028489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1028562Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1028789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1028860Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1029086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1029159Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1029385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1029487Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1029490Z 
2025-09-07T09:16:27.1029578Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1029763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1029852Z     return mod(**inputs)
2025-09-07T09:16:27.1030089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1030153Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1030385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1030446Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1030655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1030725Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1030951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1031026Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1031252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1031327Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1031560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1031654Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1031658Z 
2025-09-07T09:16:27.1031746Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1031927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1031988Z     return mod(**inputs)
2025-09-07T09:16:27.1032226Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1032288Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1032520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1032580Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1032794Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1032895Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1033123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1033202Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1033429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1033504Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1033726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1033850Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1033853Z 
2025-09-07T09:16:27.1033942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1034130Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1034195Z     return mod(**inputs)
2025-09-07T09:16:27.1034425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1034491Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1034722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1034783Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1035031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1035098Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1035328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1035429Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1035657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1035736Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1035960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1036055Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1036058Z 
2025-09-07T09:16:27.1036143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1036326Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1036385Z     return mod(**inputs)
2025-09-07T09:16:27.1036615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1036679Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1036909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1036978Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1037189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1037255Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1037483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1037551Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1037784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1037854Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1038080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1038150Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1038154Z 
2025-09-07T09:16:27.1038225Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1038352Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1038536Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1038594Z     return mod(**inputs)
2025-09-07T09:16:27.1038827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1038889Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1039122Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1039182Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1039431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1039498Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1039725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1039810Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1040036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1040124Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1040349Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1040447Z     return self.weight * hidden_states
2025-09-07T09:16:27.1040451Z 
2025-09-07T09:16:27.1040543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1040728Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1040786Z     return mod(**inputs)
2025-09-07T09:16:27.1041061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1041126Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1041359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1041420Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1041632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1041701Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1041930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1042008Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1042237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1042348Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1042571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1042646Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1042649Z 
2025-09-07T09:16:27.1042738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1042920Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1042978Z     return mod(**inputs)
2025-09-07T09:16:27.1043208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1043273Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1043502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1043564Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1043784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1043893Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1044124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1044201Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1044429Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1044533Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1044761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1044871Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1044874Z 
2025-09-07T09:16:27.1044964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1045152Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1045209Z     return mod(**inputs)
2025-09-07T09:16:27.1045437Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1045502Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1045730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1045797Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1046038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1046111Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1046336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1046417Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1046677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1046782Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1047011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1047080Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1047083Z 
2025-09-07T09:16:27.1047171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1047360Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1047416Z     return mod(**inputs)
2025-09-07T09:16:27.1047653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1047715Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1047948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1048011Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1048223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1048294Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1048521Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1048596Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1048824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:16:27.1048946Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:16:27.1048950Z 
2025-09-07T09:16:27.1049024Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1049114Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1049300Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1049403Z     return mod(**inputs)
2025-09-07T09:16:27.1049634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1049702Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1049931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1049993Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1050209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1050274Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1050543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1050616Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1050849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1050948Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1051181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1051247Z     return self.weight * hidden_states
2025-09-07T09:16:27.1051250Z 
2025-09-07T09:16:27.1051338Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1051558Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1051618Z     return mod(**inputs)
2025-09-07T09:16:27.1051854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1051918Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1052179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1052242Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1052452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1052524Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1052750Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1052825Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1053052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1053124Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1053355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1053423Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1053426Z 
2025-09-07T09:16:27.1053515Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1053700Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1053755Z     return mod(**inputs)
2025-09-07T09:16:27.1053986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1054046Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1054286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1054345Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1054555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1054626Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1054852Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1054959Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1055188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1055259Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1055488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1055555Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1055558Z 
2025-09-07T09:16:27.1055652Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1055834Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1055925Z     return mod(**inputs)
2025-09-07T09:16:27.1056157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1056220Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1056452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1056512Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1056725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1056790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1057019Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1057121Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1057354Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1057431Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1057700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1057824Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1057827Z 
2025-09-07T09:16:27.1057919Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1058177Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1058240Z     return mod(**inputs)
2025-09-07T09:16:27.1058477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1058546Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1058774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1058841Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1059062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1059133Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1059365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1059437Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1059663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1059739Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1059965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1060113Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1060116Z 
2025-09-07T09:16:27.1060206Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1060401Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1060461Z     return mod(**inputs)
2025-09-07T09:16:27.1060692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1060799Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1061030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1061094Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1061308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1061377Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1061605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1061715Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1061946Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1062016Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1062245Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1062314Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1062317Z 
2025-09-07T09:16:27.1062406Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1062596Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1062653Z     return mod(**inputs)
2025-09-07T09:16:27.1062934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1062998Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1063234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1063332Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1063545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1063619Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1063848Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1063918Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1064152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1064223Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1064452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1064553Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1064556Z 
2025-09-07T09:16:27.1064650Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1064835Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1064893Z     return mod(**inputs)
2025-09-07T09:16:27.1065125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1065186Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1065417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1065478Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1065690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1065760Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1065988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1066064Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1066288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1066393Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1066625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1066719Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1066723Z 
2025-09-07T09:16:27.1066814Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1067000Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1067058Z     return mod(**inputs)
2025-09-07T09:16:27.1067286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1067382Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1067615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1067677Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1067893Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1067959Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1068183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1068255Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1068510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1068583Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1068810Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1068935Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1068938Z 
2025-09-07T09:16:27.1069029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1069216Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1069280Z     return mod(**inputs)
2025-09-07T09:16:27.1069522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1069584Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1069816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1069876Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1070089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1070156Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1070384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1070454Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1070679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1070750Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1070973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1071051Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1071054Z 
2025-09-07T09:16:27.1071129Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1071221Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1071414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1071471Z     return mod(**inputs)
2025-09-07T09:16:27.1071712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1071812Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1072047Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1072107Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1072317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1072386Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1072616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1072691Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1072957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1073056Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1073286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1073356Z     return self.weight * hidden_states
2025-09-07T09:16:27.1073359Z 
2025-09-07T09:16:27.1073448Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1073631Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1073688Z     return mod(**inputs)
2025-09-07T09:16:27.1073978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1074041Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1074276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1074338Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1074585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1074652Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1074878Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1074955Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1075184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1075262Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1075489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1075557Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1075569Z 
2025-09-07T09:16:27.1075660Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1075844Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1075905Z     return mod(**inputs)
2025-09-07T09:16:27.1076142Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1076209Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1076435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1076494Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1076714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1076781Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1077010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1077080Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1077306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1077419Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1077644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1077717Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1077721Z 
2025-09-07T09:16:27.1077807Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1077989Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1078045Z     return mod(**inputs)
2025-09-07T09:16:27.1078273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1078393Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1078625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1078687Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1078902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1078968Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1079199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1079269Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1079541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1079612Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1079837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1079959Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1079997Z 
2025-09-07T09:16:27.1080086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1080276Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1080333Z     return mod(**inputs)
2025-09-07T09:16:27.1080567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1080630Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1080860Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1080929Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1081140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1081210Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1081438Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1081508Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1081738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1081810Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1082040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1082184Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1082187Z 
2025-09-07T09:16:27.1082281Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1082464Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1082521Z     return mod(**inputs)
2025-09-07T09:16:27.1082758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1082821Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1083106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1083166Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1083379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1083447Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1083678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1083753Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1083979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1084102Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1084334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1084405Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1084409Z 
2025-09-07T09:16:27.1084501Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1084683Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1084743Z     return mod(**inputs)
2025-09-07T09:16:27.1084973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1085034Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1085307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1085373Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1085591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1085687Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1085918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1085996Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1086223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1086295Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1086520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1086618Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1086624Z 
2025-09-07T09:16:27.1086714Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1086897Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1086956Z     return mod(**inputs)
2025-09-07T09:16:27.1087185Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1087256Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1087489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1087552Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1087765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1087829Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1088060Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1088129Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1088358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1088434Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1088697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1088792Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1088795Z 
2025-09-07T09:16:27.1088885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1089069Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1089125Z     return mod(**inputs)
2025-09-07T09:16:27.1089353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1089418Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1089684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1089752Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1089962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1090030Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1090261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1090332Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1090560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1090666Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1090894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1090988Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1090991Z 
2025-09-07T09:16:27.1091113Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1091298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1091356Z     return mod(**inputs)
2025-09-07T09:16:27.1091587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1091647Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1091875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1091937Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1092148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1092214Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1092439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1092511Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1092741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1092813Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1093039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1093105Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1093108Z 
2025-09-07T09:16:27.1093178Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1093267Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1093451Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1093508Z     return mod(**inputs)
2025-09-07T09:16:27.1093739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1093803Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1094030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1094128Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1094341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1094405Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1094632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1094714Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1094937Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1095057Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1095284Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1095354Z     return self.weight * hidden_states
2025-09-07T09:16:27.1095358Z 
2025-09-07T09:16:27.1095444Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1095625Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1095682Z     return mod(**inputs)
2025-09-07T09:16:27.1095912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1095975Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1096238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1096302Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1096515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1096612Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1096845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1096925Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1097151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1097261Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1097485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1097557Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1097560Z 
2025-09-07T09:16:27.1097648Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1097833Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1097889Z     return mod(**inputs)
2025-09-07T09:16:27.1098150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1098216Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1098443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1098506Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1098715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1098781Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1099007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1099083Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1099313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1099416Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1099643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1099752Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1099755Z 
2025-09-07T09:16:27.1099842Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1100026Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1100082Z     return mod(**inputs)
2025-09-07T09:16:27.1100316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1100377Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1100607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1100702Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1100914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1100983Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1101208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1101284Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1101508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1101607Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1101869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1101941Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1101944Z 
2025-09-07T09:16:27.1102013Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1102132Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1102316Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1102375Z     return mod(**inputs)
2025-09-07T09:16:27.1102604Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1102668Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1102896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1102958Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1103298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1103364Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1103594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1103665Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1103892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1103988Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1104213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1104281Z     return self.weight * hidden_states
2025-09-07T09:16:27.1104284Z 
2025-09-07T09:16:27.1104376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1104573Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1104632Z     return mod(**inputs)
2025-09-07T09:16:27.1104864Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1104927Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1105155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1105311Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1105522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1105588Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1105819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1105887Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1106115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1106232Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1106461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1106530Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1106534Z 
2025-09-07T09:16:27.1106623Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1106810Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1106865Z     return mod(**inputs)
2025-09-07T09:16:27.1107096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1107157Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1107435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1107498Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1107711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1107782Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1108052Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1108126Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1108350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1108420Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1108648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1108715Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1108721Z 
2025-09-07T09:16:27.1108812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1108994Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1109051Z     return mod(**inputs)
2025-09-07T09:16:27.1109287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1109349Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1109580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1109640Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1109850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1109917Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1110143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1110215Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1110440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1110513Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1110737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1110888Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1110892Z 
2025-09-07T09:16:27.1110982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1111163Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1111221Z     return mod(**inputs)
2025-09-07T09:16:27.1111450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1111512Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1111742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1111835Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1112048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1112114Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1112342Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1112410Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1112635Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1112706Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1112963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1113108Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1113113Z 
2025-09-07T09:16:27.1113199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1113412Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1113472Z     return mod(**inputs)
2025-09-07T09:16:27.1113706Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1113770Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1113999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1114059Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1114273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1114340Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1114567Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1114637Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1114868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1114940Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1115165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1115238Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1115241Z 
2025-09-07T09:16:27.1115327Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1115512Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1115573Z     return mod(**inputs)
2025-09-07T09:16:27.1115802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1115871Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1116102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1116168Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1116424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1116494Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1116725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1116794Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1117025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1117098Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1117328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1117457Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1117460Z 
2025-09-07T09:16:27.1117549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1117737Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1117794Z     return mod(**inputs)
2025-09-07T09:16:27.1118029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1118091Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1118321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1118416Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1118630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1118702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1118963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1119036Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1119264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1119333Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1119561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1119652Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1119655Z 
2025-09-07T09:16:27.1119746Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1119927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1119983Z     return mod(**inputs)
2025-09-07T09:16:27.1120217Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1120279Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1120513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1120574Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1120782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1120852Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1121078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1121150Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1121373Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1121447Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1121671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1121761Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1121797Z 
2025-09-07T09:16:27.1121888Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1122071Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1122128Z     return mod(**inputs)
2025-09-07T09:16:27.1122356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1122419Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1122650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1122711Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1122963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1123031Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1123256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1123324Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1123547Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1123618Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1123842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1123941Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1123945Z 
2025-09-07T09:16:27.1124032Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1124215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1124272Z     return mod(**inputs)
2025-09-07T09:16:27.1124546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1124612Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1124842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1124905Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1125116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1125182Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1125411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1125481Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1125709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.1125834Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.1125837Z 
2025-09-07T09:16:27.1125910Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1126003Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1126185Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1126243Z     return mod(**inputs)
2025-09-07T09:16:27.1126472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1126532Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1126765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1126827Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1127040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1127107Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1127331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1127437Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1127662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1127759Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1127984Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1128054Z     return self.weight * hidden_states
2025-09-07T09:16:27.1128057Z 
2025-09-07T09:16:27.1128146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1128359Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1128418Z     return mod(**inputs)
2025-09-07T09:16:27.1128653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1128720Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1128951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1129015Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1129232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1129298Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1129767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1129839Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1130069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1130179Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1130409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1130485Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1130488Z 
2025-09-07T09:16:27.1130575Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1130766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1130824Z     return mod(**inputs)
2025-09-07T09:16:27.1131054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1131119Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1131350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1131417Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1131630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1131695Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1131921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1131992Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1132220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1132290Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1132516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1132586Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1132591Z 
2025-09-07T09:16:27.1132678Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1132867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1132924Z     return mod(**inputs)
2025-09-07T09:16:27.1133193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1133254Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1133484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1133547Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1133758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1133829Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1134055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1134156Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1134384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1134459Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1134687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1134802Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1134805Z 
2025-09-07T09:16:27.1134898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1135079Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1135168Z     return mod(**inputs)
2025-09-07T09:16:27.1135401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1135465Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1135730Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1135791Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1136006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1136075Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1136301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1136372Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1136599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1136673Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1136901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1137045Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1137049Z 
2025-09-07T09:16:27.1137145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1137328Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1137386Z     return mod(**inputs)
2025-09-07T09:16:27.1137614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1137674Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1137906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1137966Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1138251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1138320Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1138549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1138664Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1138890Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1138966Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1139190Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1139261Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1139267Z 
2025-09-07T09:16:27.1139362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1139545Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1139637Z     return mod(**inputs)
2025-09-07T09:16:27.1139871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1139940Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1140171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1140232Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1140448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1140513Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1140746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1140846Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1141073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1141151Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1141409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1141513Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1141516Z 
2025-09-07T09:16:27.1141608Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1141792Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1141849Z     return mod(**inputs)
2025-09-07T09:16:27.1142080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1142146Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1142376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1142442Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1142654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1142724Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1142956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1143025Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1143260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1143331Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1143555Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1143653Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1143656Z 
2025-09-07T09:16:27.1143747Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1143936Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1143993Z     return mod(**inputs)
2025-09-07T09:16:27.1144229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1144326Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1144557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1144625Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1144836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1144908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1145135Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1145237Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1145469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1145542Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1145777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1145874Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1145877Z 
2025-09-07T09:16:27.1145974Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1146158Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1146213Z     return mod(**inputs)
2025-09-07T09:16:27.1146496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1146560Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1146800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1146893Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1147110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1147182Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1147412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1147484Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1147710Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1147781Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1148008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1148080Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1148083Z 
2025-09-07T09:16:27.1148157Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1148247Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1148431Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1148492Z     return mod(**inputs)
2025-09-07T09:16:27.1148721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1148788Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1149018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1149083Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1149296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1149364Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1149596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1149680Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1149951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1150042Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1150270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1150341Z     return self.weight * hidden_states
2025-09-07T09:16:27.1150344Z 
2025-09-07T09:16:27.1150435Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1150619Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1150679Z     return mod(**inputs)
2025-09-07T09:16:27.1150951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1151015Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1151249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1151319Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1151544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1151615Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1151840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1151949Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1152183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1152296Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1152559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1152630Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1152635Z 
2025-09-07T09:16:27.1152725Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1152913Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1152969Z     return mod(**inputs)
2025-09-07T09:16:27.1153204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1153268Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1153501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1153562Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1153777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1153852Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1154076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1154156Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1154382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1154482Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1154711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1154784Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1154787Z 
2025-09-07T09:16:27.1154879Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1155063Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1155123Z     return mod(**inputs)
2025-09-07T09:16:27.1155352Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1155452Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1155690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1155752Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1155965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1156032Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1156259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1156341Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1156605Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1156710Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1156940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1157007Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1157013Z 
2025-09-07T09:16:27.1157082Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1157170Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1157356Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1157414Z     return mod(**inputs)
2025-09-07T09:16:27.1157677Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1157742Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1157973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1158075Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1158289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1158361Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1158587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1158658Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1158885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1158983Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1159214Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1159284Z     return self.weight * hidden_states
2025-09-07T09:16:27.1159287Z 
2025-09-07T09:16:27.1159376Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1159564Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1159622Z     return mod(**inputs)
2025-09-07T09:16:27.1159853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1159915Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1160149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1160211Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1160421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1160491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1160721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1160794Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1161020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1161134Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1161367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1161435Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1161438Z 
2025-09-07T09:16:27.1161530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1161713Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1161770Z     return mod(**inputs)
2025-09-07T09:16:27.1162002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1162154Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1162390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1162455Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1162668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1162734Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1162962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1163037Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1163296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1163371Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1163598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1163696Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1163699Z 
2025-09-07T09:16:27.1163793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1163976Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1164038Z     return mod(**inputs)
2025-09-07T09:16:27.1164266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1164329Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1164562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1164623Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1164836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1164902Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1165133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1165204Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1165428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1165501Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1165727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1165846Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1165851Z 
2025-09-07T09:16:27.1165938Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1166121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1166183Z     return mod(**inputs)
2025-09-07T09:16:27.1166414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1166512Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1166740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1166805Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1167015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1167081Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1167313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1167381Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1167607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1167741Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1167971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1168120Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1168124Z 
2025-09-07T09:16:27.1168212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1168396Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1168451Z     return mod(**inputs)
2025-09-07T09:16:27.1168721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1168789Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1169020Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1169084Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1169334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1169407Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1169633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1169704Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1169932Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1170005Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1170238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1170307Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1170311Z 
2025-09-07T09:16:27.1170401Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1170591Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1170648Z     return mod(**inputs)
2025-09-07T09:16:27.1170884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1170949Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1171178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1171247Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1171460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1171534Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1171761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1171836Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1172064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1172170Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1172399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1172498Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1172501Z 
2025-09-07T09:16:27.1172597Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1172779Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1172837Z     return mod(**inputs)
2025-09-07T09:16:27.1173074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1173171Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1173408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1173487Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1173711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1173779Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1174004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1174079Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1174341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1174417Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1174645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1174738Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1174741Z 
2025-09-07T09:16:27.1174868Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1175056Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1175120Z     return mod(**inputs)
2025-09-07T09:16:27.1175351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1175417Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1175651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1175713Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1175930Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1175999Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1176235Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1176304Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1176535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1176617Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1176853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1176948Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1176951Z 
2025-09-07T09:16:27.1177039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1177223Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1177282Z     return mod(**inputs)
2025-09-07T09:16:27.1177511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1177581Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1177809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1177905Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1178157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1178224Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1178456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1178526Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1178761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1178865Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1179096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1179168Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1179173Z 
2025-09-07T09:16:27.1179245Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1179337Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1179520Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1179577Z     return mod(**inputs)
2025-09-07T09:16:27.1179813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1179914Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1180153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1180215Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1180457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1180529Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1180757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1180831Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1181057Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1181156Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1181384Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1181451Z     return self.weight * hidden_states
2025-09-07T09:16:27.1181454Z 
2025-09-07T09:16:27.1181548Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1181743Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1181806Z     return mod(**inputs)
2025-09-07T09:16:27.1182039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1182104Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1182340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1182402Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1182615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1182680Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1182908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1182979Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1183206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1183282Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1183550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1183619Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1183622Z 
2025-09-07T09:16:27.1183711Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1183894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1183951Z     return mod(**inputs)
2025-09-07T09:16:27.1184181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1184246Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1184512Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1184575Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1184788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1184856Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1185086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1185154Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1185382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1185454Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1185711Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1185782Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1185786Z 
2025-09-07T09:16:27.1185872Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1186093Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1186151Z     return mod(**inputs)
2025-09-07T09:16:27.1186385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1186452Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1186685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1186749Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1186960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1187029Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1187261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1187330Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1187559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1187634Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1187862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1187976Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1187980Z 
2025-09-07T09:16:27.1188070Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1188258Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1188314Z     return mod(**inputs)
2025-09-07T09:16:27.1188550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1188615Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1188844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1188961Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1189173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1189241Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1189467Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1189534Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1189767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1189841Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1190107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1190251Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1190254Z 
2025-09-07T09:16:27.1190353Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1190538Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1190596Z     return mod(**inputs)
2025-09-07T09:16:27.1190830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1190891Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1191156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1191221Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1191430Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1191506Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1191764Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1191842Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1192074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1192149Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1192374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1192444Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1192448Z 
2025-09-07T09:16:27.1192543Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1192725Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1192788Z     return mod(**inputs)
2025-09-07T09:16:27.1193017Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1193078Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1193312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1193371Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1193582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1193647Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1193875Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1193946Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1194168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1194247Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1194473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1194606Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1194609Z 
2025-09-07T09:16:27.1213008Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1213275Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1213341Z     return mod(**inputs)
2025-09-07T09:16:27.1213617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1213697Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1213945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1214134Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1214365Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1214449Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1214702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1214781Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1215025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1215102Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1215400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1215505Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1215511Z 
2025-09-07T09:16:27.1215621Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1215871Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1215939Z     return mod(**inputs)
2025-09-07T09:16:27.1216183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1216252Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1216488Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1216553Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1216778Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1216854Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1217084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1217163Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1217392Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1217471Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1217695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1217800Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1217808Z 
2025-09-07T09:16:27.1217909Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1218158Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1218221Z     return mod(**inputs)
2025-09-07T09:16:27.1218461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1218534Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1218765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1218829Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1219114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1219187Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1219425Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1219496Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1219723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1219806Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1220030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1220403Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1220407Z 
2025-09-07T09:16:27.1220509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1220699Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1220765Z     return mod(**inputs)
2025-09-07T09:16:27.1220996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1221065Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1221293Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1221358Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1221610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1221685Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1221921Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1222047Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1222287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:16:27.1222416Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.1222420Z 
2025-09-07T09:16:27.1222493Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1222590Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1222774Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1222835Z     return mod(**inputs)
2025-09-07T09:16:27.1223066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1223133Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1223368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1223434Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1223653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1223722Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1223951Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1224036Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1224264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1224356Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1224582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1224656Z     return self.weight * hidden_states
2025-09-07T09:16:27.1224660Z 
2025-09-07T09:16:27.1224756Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1224947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1225042Z     return mod(**inputs)
2025-09-07T09:16:27.1225280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1225351Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1225583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1225648Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1225863Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1225969Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1226201Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1226282Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1226510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1226621Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1226847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1226923Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1226926Z 
2025-09-07T09:16:27.1227017Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1227236Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1227295Z     return mod(**inputs)
2025-09-07T09:16:27.1227525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1227624Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1227855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1227921Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1228134Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1228206Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1228433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1228510Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1228739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1228843Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1229073Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1229148Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1229152Z 
2025-09-07T09:16:27.1229241Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1229427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1229484Z     return mod(**inputs)
2025-09-07T09:16:27.1229717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1229780Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1230009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1230073Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1230285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1230358Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1230583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1230696Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1230924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1231025Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1231256Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1231326Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1231330Z 
2025-09-07T09:16:27.1231405Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1231526Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1231711Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1231772Z     return mod(**inputs)
2025-09-07T09:16:27.1232003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1232072Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1232304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1232368Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1232584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1232702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1232938Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1233014Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1233275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1233374Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1233601Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1233675Z     return self.weight * hidden_states
2025-09-07T09:16:27.1233678Z 
2025-09-07T09:16:27.1233781Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1233983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1234043Z     return mod(**inputs)
2025-09-07T09:16:27.1234296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1234367Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1234599Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1234665Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1234883Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1234958Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1235186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1235258Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1235486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1235562Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1235790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1235860Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1235864Z 
2025-09-07T09:16:27.1235959Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1236148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1236252Z     return mod(**inputs)
2025-09-07T09:16:27.1236487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1236551Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1236780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1236845Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1237061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1237134Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1237400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1237475Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1237703Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1237778Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1238008Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1238075Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1238078Z 
2025-09-07T09:16:27.1238172Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1238390Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1238447Z     return mod(**inputs)
2025-09-07T09:16:27.1238680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1238745Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1239011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1239075Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1239288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1239358Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1239582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1239653Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1239880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1239951Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1240176Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1240301Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1240304Z 
2025-09-07T09:16:27.1240400Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1240586Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1240646Z     return mod(**inputs)
2025-09-07T09:16:27.1240884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1240948Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1241188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1241250Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1241465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1241535Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1241762Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1241872Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1242099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1242172Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1242395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1242545Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1242549Z 
2025-09-07T09:16:27.1242637Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1242821Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1242912Z     return mod(**inputs)
2025-09-07T09:16:27.1243144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1243211Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1243439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1243500Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1243714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1243781Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1244046Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1244115Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1244344Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1244445Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1244673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1244746Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1244750Z 
2025-09-07T09:16:27.1244837Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1245021Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1245076Z     return mod(**inputs)
2025-09-07T09:16:27.1245306Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1245370Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1245596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1245660Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1245872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1245939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1246166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1246234Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1246459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1246528Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1246758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1246859Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1246864Z 
2025-09-07T09:16:27.1246952Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1247139Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1247195Z     return mod(**inputs)
2025-09-07T09:16:27.1247466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1247528Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1247756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1247819Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1248031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1248104Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1248328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1248437Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1248667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1248737Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1248966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1249059Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1249062Z 
2025-09-07T09:16:27.1249151Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1249332Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1249419Z     return mod(**inputs)
2025-09-07T09:16:27.1249653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1249717Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1249980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1250041Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1250254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1250322Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1250552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1250625Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1250854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1250925Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1251150Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1251244Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1251247Z 
2025-09-07T09:16:27.1251338Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1251521Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1251578Z     return mod(**inputs)
2025-09-07T09:16:27.1251806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1251868Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1252097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1252157Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1252370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1252438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1252664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1252736Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1252999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1253070Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1253295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1253363Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1253367Z 
2025-09-07T09:16:27.1253440Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1253530Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1253714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1253821Z     return mod(**inputs)
2025-09-07T09:16:27.1254054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1254116Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1254346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1254411Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1254620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1254689Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1254913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1255026Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1255259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1255359Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1255621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1255691Z     return self.weight * hidden_states
2025-09-07T09:16:27.1255694Z 
2025-09-07T09:16:27.1255785Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1255969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1256024Z     return mod(**inputs)
2025-09-07T09:16:27.1256257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1256321Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1256554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1256617Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1256830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1256899Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1257127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1257199Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1257423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1257495Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1257721Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1257790Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1257793Z 
2025-09-07T09:16:27.1257882Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1258104Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1258164Z     return mod(**inputs)
2025-09-07T09:16:27.1258394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1258494Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1258724Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1258784Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1258996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1259061Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1259287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1259391Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1259618Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1259693Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1259917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1259983Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1259989Z 
2025-09-07T09:16:27.1260078Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1260259Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1260316Z     return mod(**inputs)
2025-09-07T09:16:27.1260575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1260640Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1260872Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1260933Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1261181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1261251Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1261481Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1261551Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1261777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1261851Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1262078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1262204Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1262208Z 
2025-09-07T09:16:27.1262299Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1262484Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1262542Z     return mod(**inputs)
2025-09-07T09:16:27.1262772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1262837Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1263064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1263130Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1263343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1263409Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1263638Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1263711Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1263940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1264048Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1264273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1264420Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1264423Z 
2025-09-07T09:16:27.1264512Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1264701Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1264756Z     return mod(**inputs)
2025-09-07T09:16:27.1264992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1265089Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1265319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1265385Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1265596Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1265666Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1265892Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1265961Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1266223Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1266301Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1266533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1266634Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1266638Z 
2025-09-07T09:16:27.1266740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1266927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1266986Z     return mod(**inputs)
2025-09-07T09:16:27.1267220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1267280Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1267515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1267576Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1267787Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1267864Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1268092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1268171Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1268396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1268466Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1268698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1268793Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1268797Z 
2025-09-07T09:16:27.1268891Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1269074Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1269136Z     return mod(**inputs)
2025-09-07T09:16:27.1269367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1269429Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1269699Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1269762Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1269976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1270040Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1270271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1270345Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1270570Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1270679Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1270906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1270998Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1271006Z 
2025-09-07T09:16:27.1271096Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1271277Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1271336Z     return mod(**inputs)
2025-09-07T09:16:27.1271564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1271663Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1271896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1271961Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1272206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1272273Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1272504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1272572Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1272797Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1272872Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1273101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1273197Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1273202Z 
2025-09-07T09:16:27.1273289Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1273476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1273532Z     return mod(**inputs)
2025-09-07T09:16:27.1273763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1273827Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1274055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1274116Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1274326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1274395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1274625Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1274696Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1274924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1275044Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1275271Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1275344Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1275347Z 
2025-09-07T09:16:27.1275416Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1275506Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1275688Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1275750Z     return mod(**inputs)
2025-09-07T09:16:27.1275978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1276072Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1276309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1276371Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1276586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1276650Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1276877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1276966Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1277224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1277313Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1277542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1277610Z     return self.weight * hidden_states
2025-09-07T09:16:27.1277649Z 
2025-09-07T09:16:27.1277741Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1277924Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1277987Z     return mod(**inputs)
2025-09-07T09:16:27.1278215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1278282Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1278511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1278572Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1278808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1278878Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1279110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1279189Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1279418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1279530Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1279756Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1279831Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1279834Z 
2025-09-07T09:16:27.1279924Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1280112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1280172Z     return mod(**inputs)
2025-09-07T09:16:27.1280401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1280471Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1280738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1280805Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1281016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1281083Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1281312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1281389Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1281615Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1281749Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1281979Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1282055Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1282058Z 
2025-09-07T09:16:27.1282146Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1282333Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1282390Z     return mod(**inputs)
2025-09-07T09:16:27.1282622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1282714Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1282947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1283019Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1283262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1283333Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1283563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1283639Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1283871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1283972Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1284200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1284269Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1284272Z 
2025-09-07T09:16:27.1284362Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1284547Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1284604Z     return mod(**inputs)
2025-09-07T09:16:27.1284836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1284898Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1285130Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1285191Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1285401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1285471Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1285695Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1285772Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1285999Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward
2025-09-07T09:16:27.1286119Z     hidden_states = hidden_states + self.dropout(forwarded_states)
2025-09-07T09:16:27.1286155Z 
2025-09-07T09:16:27.1286229Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1286319Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1286503Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1286558Z     return mod(**inputs)
2025-09-07T09:16:27.1286788Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1286855Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1287084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1287182Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1287395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1287466Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1287690Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1287763Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1287992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1288086Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1288348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1288413Z     return self.weight * hidden_states
2025-09-07T09:16:27.1288418Z 
2025-09-07T09:16:27.1288505Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1288720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1288776Z     return mod(**inputs)
2025-09-07T09:16:27.1289012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1289079Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1289313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1289372Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1289582Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1289655Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1289882Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1289960Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1290188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1290260Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1290492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1290560Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1290563Z 
2025-09-07T09:16:27.1290656Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1290835Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1290893Z     return mod(**inputs)
2025-09-07T09:16:27.1291125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1291188Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1291421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1291483Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1291697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1291800Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1292026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1292098Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1292321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1292398Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1292622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1292721Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1292724Z 
2025-09-07T09:16:27.1292817Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1292998Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1293061Z     return mod(**inputs)
2025-09-07T09:16:27.1293290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1293355Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1293586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1293646Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1293891Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1293957Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1294191Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1294307Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1294538Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1294616Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1294839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1294962Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1294966Z 
2025-09-07T09:16:27.1295054Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1295235Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1295293Z     return mod(**inputs)
2025-09-07T09:16:27.1295525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1295592Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1295821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1295885Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1296100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1296165Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1296399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1296470Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1296701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1296770Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1296995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1297145Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1297181Z 
2025-09-07T09:16:27.1297271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1297457Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1297512Z     return mod(**inputs)
2025-09-07T09:16:27.1297743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1297808Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1298087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1298150Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1298397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1298472Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1298700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1298773Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1299002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1299071Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1299298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1299367Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1299402Z 
2025-09-07T09:16:27.1299493Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1299681Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1299738Z     return mod(**inputs)
2025-09-07T09:16:27.1300005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1300072Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1300305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1300370Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1300581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1300652Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1300879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1300952Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1301178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1301248Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1301478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1301576Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1301580Z 
2025-09-07T09:16:27.1301674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1301855Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1301910Z     return mod(**inputs)
2025-09-07T09:16:27.1302141Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1302204Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1302435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1302500Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1302713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1302817Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1303232Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1303303Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1303532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1303609Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1303837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1303929Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1303993Z 
2025-09-07T09:16:27.1304086Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1304269Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1304327Z     return mod(**inputs)
2025-09-07T09:16:27.1304560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1304623Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1304857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1304920Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1305184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1305253Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1305486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1305556Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1305823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1305899Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1306126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1306221Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1306225Z 
2025-09-07T09:16:27.1306314Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1306497Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1306557Z     return mod(**inputs)
2025-09-07T09:16:27.1306789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1306857Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1307086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1307148Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1307362Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1307427Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1307655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1307722Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1307952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1308022Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1308247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1308317Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1308320Z 
2025-09-07T09:16:27.1308392Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1308484Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1308717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1308775Z     return mod(**inputs)
2025-09-07T09:16:27.1309014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1309078Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1309313Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1309376Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1309593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1309701Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1309929Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1310005Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1310230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1310333Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1310563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1310630Z     return self.weight * hidden_states
2025-09-07T09:16:27.1310633Z 
2025-09-07T09:16:27.1310782Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1310966Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1311030Z     return mod(**inputs)
2025-09-07T09:16:27.1311287Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1311353Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1311589Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1311649Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1311861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1311927Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1312154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1312231Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1312458Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1312538Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1312767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1312840Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1312843Z 
2025-09-07T09:16:27.1312930Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1313113Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1313172Z     return mod(**inputs)
2025-09-07T09:16:27.1313401Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1313466Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1313696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1313758Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1313974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1314040Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1314309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1314378Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1314606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1314682Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1314905Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1314975Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1314978Z 
2025-09-07T09:16:27.1315070Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1315290Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1315347Z     return mod(**inputs)
2025-09-07T09:16:27.1315574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1315643Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1315873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1315938Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1316148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1316215Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1316491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1316562Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1316791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1316895Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1317127Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1317248Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1317252Z 
2025-09-07T09:16:27.1317343Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1317529Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1317584Z     return mod(**inputs)
2025-09-07T09:16:27.1317820Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1317883Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1318114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1318178Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1318388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1318460Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1318684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1318755Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1318985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1319059Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1319286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1319431Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1319435Z 
2025-09-07T09:16:27.1319529Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1319710Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1319801Z     return mod(**inputs)
2025-09-07T09:16:27.1320038Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1320103Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1320338Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1320399Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1320616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1320689Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1320956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1321034Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1321259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1321333Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1321565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1321636Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1321639Z 
2025-09-07T09:16:27.1321732Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1321951Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1322015Z     return mod(**inputs)
2025-09-07T09:16:27.1322243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1322308Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1322579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1322643Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1322862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1322928Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1323153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1323228Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1323456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1323533Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1323761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1323867Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1323870Z 
2025-09-07T09:16:27.1323965Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1324148Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1324208Z     return mod(**inputs)
2025-09-07T09:16:27.1324434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1324502Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1324731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1324794Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1325007Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1325073Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1325304Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1325428Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1325655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1325731Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1325956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1326058Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1326063Z 
2025-09-07T09:16:27.1326152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1326336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1326426Z     return mod(**inputs)
2025-09-07T09:16:27.1326660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1326729Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1326958Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1327025Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1327234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1327299Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1327562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1327635Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1327866Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1327938Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1328192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1328295Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1328298Z 
2025-09-07T09:16:27.1328388Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1328571Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1328625Z     return mod(**inputs)
2025-09-07T09:16:27.1328858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1328921Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1329153Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1329221Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1329434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1329504Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1329735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1329807Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1330035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1330113Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1330345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1330415Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1330419Z 
2025-09-07T09:16:27.1330494Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1330583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1330765Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1330863Z     return mod(**inputs)
2025-09-07T09:16:27.1331090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1331155Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1331383Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1331445Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1331665Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1331732Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1331962Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1332079Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1332308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1332400Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1332627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1332700Z     return self.weight * hidden_states
2025-09-07T09:16:27.1332704Z 
2025-09-07T09:16:27.1332793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1333009Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1333067Z     return mod(**inputs)
2025-09-07T09:16:27.1333298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1333375Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1333639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1333707Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1333919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1333985Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1334219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1334296Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1334531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1334640Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1334870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1334948Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1334951Z 
2025-09-07T09:16:27.1335039Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1335226Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1335283Z     return mod(**inputs)
2025-09-07T09:16:27.1335517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1335578Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1335811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1335875Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1336084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1336155Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1336379Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1336488Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1336719Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1336820Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1337048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1337120Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1337123Z 
2025-09-07T09:16:27.1337217Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1337399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1337511Z     return mod(**inputs)
2025-09-07T09:16:27.1337747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1337811Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1338079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1338140Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1338351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1338421Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1338682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1338763Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1338992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1339095Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1339357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1339432Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1339434Z 
2025-09-07T09:16:27.1339513Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1339602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1339790Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1339847Z     return mod(**inputs)
2025-09-07T09:16:27.1340079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1340145Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1340376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1340443Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1340654Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1340724Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1340955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1341025Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1341255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1341351Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1341576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1341647Z     return self.weight * hidden_states
2025-09-07T09:16:27.1341651Z 
2025-09-07T09:16:27.1341740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1341929Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1341985Z     return mod(**inputs)
2025-09-07T09:16:27.1342261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1342327Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1342557Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1342623Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1342841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1342915Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1343139Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1343244Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1343478Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1343553Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1343782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1343851Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1343855Z 
2025-09-07T09:16:27.1343945Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1344132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1344220Z     return mod(**inputs)
2025-09-07T09:16:27.1344457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1344520Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1344786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1344850Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1345064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1345136Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1345364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1345435Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1345669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1345741Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1345975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1346044Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1346048Z 
2025-09-07T09:16:27.1346143Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1346325Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1346383Z     return mod(**inputs)
2025-09-07T09:16:27.1346617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1346680Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1346913Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1346973Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1347189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1347258Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1347485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1347560Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1347821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1347896Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1348123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1348242Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1348246Z 
2025-09-07T09:16:27.1348338Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1348523Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1348586Z     return mod(**inputs)
2025-09-07T09:16:27.1348851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1348920Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1349154Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1349216Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1349431Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1349498Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1349729Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1349829Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1350061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1350137Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1350396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1350552Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1350558Z 
2025-09-07T09:16:27.1350647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1350829Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1350887Z     return mod(**inputs)
2025-09-07T09:16:27.1351116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1351181Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1351411Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1351475Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1351685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1351752Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1351983Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1352051Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1352283Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1352351Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1352576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1352647Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1352651Z 
2025-09-07T09:16:27.1352738Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1352944Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1353001Z     return mod(**inputs)
2025-09-07T09:16:27.1353229Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1353329Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1353560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1353626Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1353836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1353905Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1354133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1354201Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1354468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1354541Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1354775Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1354871Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1354874Z 
2025-09-07T09:16:27.1354964Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1355155Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1355212Z     return mod(**inputs)
2025-09-07T09:16:27.1355480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1355542Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1355782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1355846Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1356092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1356166Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1356396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1356469Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1356696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1356769Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1357003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1357099Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1357102Z 
2025-09-07T09:16:27.1357200Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1357381Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1357443Z     return mod(**inputs)
2025-09-07T09:16:27.1357679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1357740Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1357974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1358036Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1358255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1358323Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1358549Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1358621Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1358850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1358968Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1359194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1359285Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1359288Z 
2025-09-07T09:16:27.1359378Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1359560Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1359618Z     return mod(**inputs)
2025-09-07T09:16:27.1359846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1359941Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1360178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1360239Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1360451Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1360515Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1360742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1360810Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1361067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1361142Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1361370Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1361442Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1361482Z 
2025-09-07T09:16:27.1361572Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1361754Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1361812Z     return mod(**inputs)
2025-09-07T09:16:27.1362045Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1362111Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1362341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1362407Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1362619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1362687Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1362916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1362983Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1363215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward
2025-09-07T09:16:27.1363338Z     hidden_states = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.1363342Z 
2025-09-07T09:16:27.1363412Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1363507Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1363687Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1363747Z     return mod(**inputs)
2025-09-07T09:16:27.1363974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1364037Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1364272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1364368Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1364584Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1364650Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1364880Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1364958Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1365189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1365284Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1365544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1365618Z     return self.weight * hidden_states
2025-09-07T09:16:27.1365622Z 
2025-09-07T09:16:27.1365713Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1365895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1365953Z     return mod(**inputs)
2025-09-07T09:16:27.1366184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1366249Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1366510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1366572Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1366790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1366859Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1367119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1367191Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1367426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1367500Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1367727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1367802Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1367805Z 
2025-09-07T09:16:27.1367894Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1368083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1368141Z     return mod(**inputs)
2025-09-07T09:16:27.1368371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1368436Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1368667Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1368734Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1368950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1369014Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1369243Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1369316Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1369548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1369623Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1369854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1369958Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1369961Z 
2025-09-07T09:16:27.1370048Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1370233Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1370287Z     return mod(**inputs)
2025-09-07T09:16:27.1370523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1370586Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1370816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1370916Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1371123Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1371195Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1371423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1371492Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1371725Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1371795Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1372026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1372173Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1372177Z 
2025-09-07T09:16:27.1372271Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1372454Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1372541Z     return mod(**inputs)
2025-09-07T09:16:27.1372779Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1372843Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1373074Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1373135Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1373345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1373416Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1373643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1373720Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1373945Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1374019Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1374246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1374387Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1374391Z 
2025-09-07T09:16:27.1374483Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1374665Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1374723Z     return mod(**inputs)
2025-09-07T09:16:27.1374957Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1375020Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1375255Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1375317Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1375532Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1375634Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1375859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1375930Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1376156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1376233Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1376460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1376562Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1376565Z 
2025-09-07T09:16:27.1376656Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1376839Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1376902Z     return mod(**inputs)
2025-09-07T09:16:27.1377131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1377199Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1377428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1377489Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1377736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1377805Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1378106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1378219Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1378450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1378529Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1378757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1378853Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1378857Z 
2025-09-07T09:16:27.1378944Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1379135Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1379194Z     return mod(**inputs)
2025-09-07T09:16:27.1379424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1379489Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1379720Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1379793Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1380005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1380073Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1380303Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1380375Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1380606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1380678Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1380908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1381000Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1381037Z 
2025-09-07T09:16:27.1381126Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1381318Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1381373Z     return mod(**inputs)
2025-09-07T09:16:27.1381609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1381678Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1381914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1381982Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1382194Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1382301Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1382529Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1382602Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1382833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1382905Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1383133Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1383222Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1383257Z 
2025-09-07T09:16:27.1383347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1383528Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1383584Z     return mod(**inputs)
2025-09-07T09:16:27.1383847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1383914Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1384145Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1384205Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1384416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1384485Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1384712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1384785Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1385012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1385083Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1385312Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1385382Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1385385Z 
2025-09-07T09:16:27.1385692Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1385820Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1386098Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1386185Z     return mod(**inputs)
2025-09-07T09:16:27.1386450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1386559Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1386862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1387000Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1387257Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1387386Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1387675Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1387773Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1388111Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1388242Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1388531Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1388660Z     return self.weight * hidden_states
2025-09-07T09:16:27.1388664Z 
2025-09-07T09:16:27.1388780Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1389057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1389170Z     return mod(**inputs)
2025-09-07T09:16:27.1389464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1389555Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1389816Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1389922Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1390261Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1390423Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1390708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1390849Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1391143Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1391283Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1391634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1391737Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1391741Z 
2025-09-07T09:16:27.1391893Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1392122Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1392242Z     return mod(**inputs)
2025-09-07T09:16:27.1392554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1392666Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1392959Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1393053Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1393348Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1393436Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1393749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1393901Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1394162Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1394340Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1394603Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1394790Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1394794Z 
2025-09-07T09:16:27.1394927Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1395187Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1395322Z     return mod(**inputs)
2025-09-07T09:16:27.1395595Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1395707Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1396016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1396123Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1396405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1396555Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1396853Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1396954Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1397326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1397463Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1397736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1397867Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1397871Z 
2025-09-07T09:16:27.1398008Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1398199Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1398438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1398540Z     return mod(**inputs)
2025-09-07T09:16:27.1398867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1398996Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1399274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1399406Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1399714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1399813Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1400072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1400207Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1400450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward
2025-09-07T09:16:27.1400675Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1400934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1401065Z     return self.weight * hidden_states
2025-09-07T09:16:27.1401068Z 
2025-09-07T09:16:27.1401191Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1401408Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1401568Z     return mod(**inputs)
2025-09-07T09:16:27.1401846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1401972Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1402233Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1402327Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1402590Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1402775Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1403250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1403352Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1403673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1403793Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1404059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1404240Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1404305Z 
2025-09-07T09:16:27.1404428Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1404673Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1404775Z     return mod(**inputs)
2025-09-07T09:16:27.1405041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1405198Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1405470Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1405592Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1405895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1406023Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1406269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1406416Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1406757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1406880Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1407169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1407268Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1407272Z 
2025-09-07T09:16:27.1407455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1407685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1407796Z     return mod(**inputs)
2025-09-07T09:16:27.1408101Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1408194Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1408473Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1408608Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1408867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1409010Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1409266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1409395Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1409639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1409785Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1410098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1410253Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1410257Z 
2025-09-07T09:16:27.1410410Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1410681Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1410830Z     return mod(**inputs)
2025-09-07T09:16:27.1411121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1411216Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1411508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1411604Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1411867Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1412057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1412390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1412491Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1412749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1412880Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1413121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1413421Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1413425Z 
2025-09-07T09:16:27.1413583Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1413805Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1413924Z     return mod(**inputs)
2025-09-07T09:16:27.1414200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1414384Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1414660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1414784Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1415027Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1415138Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1415416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1415558Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1415859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1415961Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1416219Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1416362Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1416367Z 
2025-09-07T09:16:27.1416475Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1416798Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1416887Z     return mod(**inputs)
2025-09-07T09:16:27.1417192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1417286Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1417551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1417702Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1417960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1418157Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1418428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1418564Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1418838Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1418977Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1419277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1419423Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1419426Z 
2025-09-07T09:16:27.1419580Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1419838Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1419914Z     return mod(**inputs)
2025-09-07T09:16:27.1420266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1420378Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1420673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1420765Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1421035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1421198Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1421517Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1421652Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1421916Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1422077Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1422325Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1422511Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1422559Z 
2025-09-07T09:16:27.1422684Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1422901Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1423018Z     return mod(**inputs)
2025-09-07T09:16:27.1423285Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1423457Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1423735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1423828Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1424105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1424206Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1424494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1424645Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1424917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1425050Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1425355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1425534Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1425538Z 
2025-09-07T09:16:27.1425654Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1425973Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1426107Z     return mod(**inputs)
2025-09-07T09:16:27.1426376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1426516Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1426781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1426940Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1427205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1427306Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1427648Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward
2025-09-07T09:16:27.1427752Z     self_attention_outputs = self.layer[0](
2025-09-07T09:16:27.1428029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward
2025-09-07T09:16:27.1428189Z     attention_output = self.SelfAttention(
2025-09-07T09:16:27.1428494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1428592Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1428596Z 
2025-09-07T09:16:27.1428713Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1428869Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1429112Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1429335Z     return mod(**inputs)
2025-09-07T09:16:27.1429645Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1429753Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1430078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1430174Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1430485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1430595Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1430900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1431003Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1431262Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward
2025-09-07T09:16:27.1431409Z     normed_hidden_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1431712Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1431868Z     return self.weight * hidden_states
2025-09-07T09:16:27.1431872Z 
2025-09-07T09:16:27.1431993Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1432240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1432328Z     return mod(**inputs)
2025-09-07T09:16:27.1432576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1432765Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1433030Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1433153Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1433395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1433494Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1433815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1434002Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1434290Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1434393Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1434680Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward
2025-09-07T09:16:27.1434767Z     query_states = self.q(hidden_states)
2025-09-07T09:16:27.1434771Z 
2025-09-07T09:16:27.1434945Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1435207Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1435329Z     return mod(**inputs)
2025-09-07T09:16:27.1435622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1435727Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1435978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1436154Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1436398Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1436526Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1436839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1436960Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1437264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1437378Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1437694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward
2025-09-07T09:16:27.1437808Z     key_states = self.k(current_states)
2025-09-07T09:16:27.1437811Z 
2025-09-07T09:16:27.1437962Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1438184Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1438311Z     return mod(**inputs)
2025-09-07T09:16:27.1438617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1438713Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1439018Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1439114Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1439415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1439523Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1439786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1439933Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1440193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1440296Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1444552Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward
2025-09-07T09:16:27.1444711Z     scores = torch.matmul(query_states, key_states.transpose(3, 2))
2025-09-07T09:16:27.1444718Z 
2025-09-07T09:16:27.1444828Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1445046Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1445108Z     return mod(**inputs)
2025-09-07T09:16:27.1445456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1445529Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1445770Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1445843Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1446067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1446153Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1446391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1446510Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1446746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1446824Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1447062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward
2025-09-07T09:16:27.1447211Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores)
2025-09-07T09:16:27.1447214Z 
2025-09-07T09:16:27.1447320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1447521Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1447622Z     return mod(**inputs)
2025-09-07T09:16:27.1447869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1447943Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1448210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1448279Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1448501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1448579Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1448808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1448885Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1449114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1449197Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1449420Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward
2025-09-07T09:16:27.1449494Z     value_states = self.v(current_states)
2025-09-07T09:16:27.1449498Z 
2025-09-07T09:16:27.1449602Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1449793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1449859Z     return mod(**inputs)
2025-09-07T09:16:27.1450092Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1450159Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1450397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1450462Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1450683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1450753Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1450982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1451057Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1451324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1451403Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1451630Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1451738Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1451741Z 
2025-09-07T09:16:27.1451835Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1452023Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1452087Z     return mod(**inputs)
2025-09-07T09:16:27.1452369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1452447Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1452694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1452760Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1452985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1453057Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1453296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1453421Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1453660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1453739Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1454003Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward
2025-09-07T09:16:27.1454109Z     attn_output = torch.matmul(attn_weights, value_states)
2025-09-07T09:16:27.1454114Z 
2025-09-07T09:16:27.1454209Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1454414Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1454473Z     return mod(**inputs)
2025-09-07T09:16:27.1454726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1454797Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1455033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1455104Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1455324Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1455395Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1455650Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1455728Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1455966Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1456043Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1456273Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward
2025-09-07T09:16:27.1456372Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:16:27.1456375Z 
2025-09-07T09:16:27.1456470Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1456671Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1456726Z     return mod(**inputs)
2025-09-07T09:16:27.1456967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1457065Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1457301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1457371Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1457585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1457659Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1457888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1457961Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1458308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward
2025-09-07T09:16:27.1458386Z     attention_output = self.EncDecAttention(
2025-09-07T09:16:27.1458619Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward
2025-09-07T09:16:27.1458694Z     attn_output = self.o(attn_output)
2025-09-07T09:16:27.1458698Z 
2025-09-07T09:16:27.1458798Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1458983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1459042Z     return mod(**inputs)
2025-09-07T09:16:27.1459310Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1459375Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1459614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1459680Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1459924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1460001Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1460237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward
2025-09-07T09:16:27.1460311Z     cross_attention_outputs = self.layer[1](
2025-09-07T09:16:27.1460546Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward
2025-09-07T09:16:27.1460679Z     layer_output = hidden_states + self.dropout(attention_output[0])
2025-09-07T09:16:27.1460684Z 
2025-09-07T09:16:27.1460762Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1460870Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1461083Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1461147Z     return mod(**inputs)
2025-09-07T09:16:27.1461399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1461472Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1461704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1461775Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1461991Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1462071Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1462307Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1462395Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1462627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward
2025-09-07T09:16:27.1462717Z     forwarded_states = self.layer_norm(hidden_states)
2025-09-07T09:16:27.1462949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward
2025-09-07T09:16:27.1463054Z     return self.weight * hidden_states
2025-09-07T09:16:27.1463057Z 
2025-09-07T09:16:27.1463154Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1463345Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1463405Z     return mod(**inputs)
2025-09-07T09:16:27.1463646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1463712Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1463947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1464045Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1464264Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1464342Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1464572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1464655Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1464885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1464997Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1465260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward
2025-09-07T09:16:27.1465335Z     hidden_states = self.wi(hidden_states)
2025-09-07T09:16:27.1465338Z 
2025-09-07T09:16:27.1465437Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1465660Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1465726Z     return mod(**inputs)
2025-09-07T09:16:27.1465960Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1466025Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1466260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1466323Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1466544Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1466613Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1466845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1466935Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1467161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1467274Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1467509Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward
2025-09-07T09:16:27.1467588Z     hidden_states = self.act(hidden_states)
2025-09-07T09:16:27.1467592Z 
2025-09-07T09:16:27.1467682Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1467868Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1467932Z     return mod(**inputs)
2025-09-07T09:16:27.1468160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward
2025-09-07T09:16:27.1468233Z     decoder_outputs = self.decoder(
2025-09-07T09:16:27.1468465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward
2025-09-07T09:16:27.1468562Z     layer_outputs = layer_module(
2025-09-07T09:16:27.1468777Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:16:27.1468849Z     return super().__call__(*args, **kwargs)
2025-09-07T09:16:27.1469082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward
2025-09-07T09:16:27.1469158Z     hidden_states = self.layer[-1](hidden_states)
2025-09-07T09:16:27.1469390Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
2025-09-07T09:16:27.1469496Z     forwarded_states = self.DenseReluDense(forwarded_states)
2025-09-07T09:16:27.1469760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward
2025-09-07T09:16:27.1469840Z     hidden_states = self.wo(hidden_states)
2025-09-07T09:16:27.1469843Z 
2025-09-07T09:16:27.1469917Z cudagraph partition due to non gpu ops
2025-09-07T09:16:27.1470010Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1470195Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1470257Z     return mod(**inputs)
2025-09-07T09:16:27.1470491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward
2025-09-07T09:16:27.1470601Z     sequence_output = sequence_output * (self.model_dim**-0.5)
2025-09-07T09:16:27.1470636Z 
2025-09-07T09:16:27.1470731Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:16:27.1470917Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:16:27.1470975Z     return mod(**inputs)
2025-09-07T09:16:27.1471238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward
2025-09-07T09:16:27.1471317Z     lm_logits = self.lm_head(sequence_output)
2025-09-07T09:16:27.1471322Z 
2025-09-07T09:16:55.2504088Z 
2025-09-07T09:16:56.1024380Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:16:56.9558522Z running benchmark:   3% 1/30 [00:00<00:24,  1.17it/s]
2025-09-07T09:16:57.8176256Z running benchmark:   7% 2/30 [00:01<00:23,  1.17it/s]
2025-09-07T09:16:58.6765287Z running benchmark:  10% 3/30 [00:02<00:23,  1.17it/s]
2025-09-07T09:16:59.5359162Z running benchmark:  13% 4/30 [00:03<00:22,  1.17it/s]
2025-09-07T09:17:00.3898949Z running benchmark:  17% 5/30 [00:04<00:21,  1.17it/s]
2025-09-07T09:17:01.2447411Z running benchmark:  20% 6/30 [00:05<00:20,  1.17it/s]
2025-09-07T09:17:02.1048571Z running benchmark:  23% 7/30 [00:05<00:19,  1.17it/s]
2025-09-07T09:17:02.9578740Z running benchmark:  27% 8/30 [00:06<00:18,  1.17it/s]
2025-09-07T09:17:03.8221568Z running benchmark:  30% 9/30 [00:07<00:17,  1.17it/s]
2025-09-07T09:17:04.6829224Z running benchmark:  33% 10/30 [00:08<00:17,  1.16it/s]
2025-09-07T09:17:05.5428552Z running benchmark:  37% 11/30 [00:09<00:16,  1.16it/s]
2025-09-07T09:17:06.4198446Z running benchmark:  40% 12/30 [00:10<00:15,  1.16it/s]
2025-09-07T09:17:07.3452329Z running benchmark:  43% 13/30 [00:11<00:14,  1.16it/s]
2025-09-07T09:17:08.1999398Z running benchmark:  47% 14/30 [00:12<00:14,  1.13it/s]
2025-09-07T09:17:09.0610690Z running benchmark:  50% 15/30 [00:12<00:13,  1.14it/s]
2025-09-07T09:17:09.9180083Z running benchmark:  53% 16/30 [00:13<00:12,  1.15it/s]
2025-09-07T09:17:10.7823102Z running benchmark:  57% 17/30 [00:14<00:11,  1.15it/s]
2025-09-07T09:17:11.6381220Z running benchmark:  60% 18/30 [00:15<00:10,  1.16it/s]
2025-09-07T09:17:12.4978565Z running benchmark:  63% 19/30 [00:16<00:09,  1.16it/s]
2025-09-07T09:17:13.3541076Z running benchmark:  67% 20/30 [00:17<00:08,  1.16it/s]
2025-09-07T09:17:14.2141883Z running benchmark:  70% 21/30 [00:18<00:07,  1.16it/s]
2025-09-07T09:17:15.0656299Z running benchmark:  73% 22/30 [00:18<00:06,  1.16it/s]
2025-09-07T09:17:15.9357026Z running benchmark:  77% 23/30 [00:19<00:06,  1.17it/s]
2025-09-07T09:17:16.7916736Z running benchmark:  80% 24/30 [00:20<00:05,  1.16it/s]
2025-09-07T09:17:17.6560827Z running benchmark:  83% 25/30 [00:21<00:04,  1.16it/s]
2025-09-07T09:17:18.5092943Z running benchmark:  87% 26/30 [00:22<00:03,  1.16it/s]
2025-09-07T09:17:19.3730178Z running benchmark:  90% 27/30 [00:23<00:02,  1.16it/s]
2025-09-07T09:17:20.2311155Z running benchmark:  93% 28/30 [00:24<00:01,  1.16it/s]
2025-09-07T09:17:21.0952326Z running benchmark:  97% 29/30 [00:24<00:00,  1.16it/s]
2025-09-07T09:17:21.0952866Z running benchmark: 100% 30/30 [00:25<00:00,  1.16it/s]
2025-09-07T09:17:21.0953473Z running benchmark: 100% 30/30 [00:25<00:00,  1.16it/s]
2025-09-07T09:17:21.3424968Z 1.471x
2025-09-07T09:17:21.3430193Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:17:24.7995564Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:17:24.7996621Z   import pynvml  # type: ignore[import]
2025-09-07T09:17:26.8116126Z 
2025-09-07T09:17:31.5321615Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:17:31.5321915Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:17:31.5646054Z cpu  eval  hf_distil_whisper                  
2025-09-07T09:17:41.7385942Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7386894Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7387238Z     return mod(*inputs)
2025-09-07T09:17:41.7387671Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7388237Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7388679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 677, in forward
2025-09-07T09:17:41.7389174Z     inputs_embeds = nn.functional.gelu(self.conv1(input_features))
2025-09-07T09:17:41.7389359Z 
2025-09-07T09:17:41.7389469Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7389892Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7390224Z     return mod(*inputs)
2025-09-07T09:17:41.7390627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7391065Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7391472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 677, in forward
2025-09-07T09:17:41.7391932Z     inputs_embeds = nn.functional.gelu(self.conv1(input_features))
2025-09-07T09:17:41.7392112Z 
2025-09-07T09:17:41.7392227Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7392579Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7392895Z     return mod(*inputs)
2025-09-07T09:17:41.7393269Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7393668Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7394055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 678, in forward
2025-09-07T09:17:41.7394578Z     inputs_embeds = nn.functional.gelu(self.conv2(inputs_embeds))
2025-09-07T09:17:41.7394757Z 
2025-09-07T09:17:41.7394854Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7395202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7395518Z     return mod(*inputs)
2025-09-07T09:17:41.7395994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7396403Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7396784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7397183Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7397528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7397905Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7398308Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 409, in forward
2025-09-07T09:17:41.7398844Z     hidden_states = self.self_attn_layer_norm(hidden_states)
2025-09-07T09:17:41.7399019Z 
2025-09-07T09:17:41.7399121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7399482Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7399814Z     return mod(*inputs)
2025-09-07T09:17:41.7400182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7400593Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7400976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7401431Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7401784Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7402148Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7402583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 409, in forward
2025-09-07T09:17:41.7403013Z     hidden_states = self.self_attn_layer_norm(hidden_states)
2025-09-07T09:17:41.7403303Z 
2025-09-07T09:17:41.7403409Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7403759Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7404072Z     return mod(*inputs)
2025-09-07T09:17:41.7404441Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7404840Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7405231Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7405619Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7405970Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7406324Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7406748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7407180Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7407591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7408019Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7408186Z 
2025-09-07T09:17:41.7408286Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7408642Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7408962Z     return mod(*inputs)
2025-09-07T09:17:41.7409327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7409754Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7410156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7410628Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7410971Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7411330Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7411722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7412179Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7412585Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7413130Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7413358Z 
2025-09-07T09:17:41.7413455Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7413808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7414136Z     return mod(*inputs)
2025-09-07T09:17:41.7414504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7414908Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7415300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7415755Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7416105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7416470Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7416902Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7417325Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7417814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7418309Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7418529Z 
2025-09-07T09:17:41.7418613Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7418812Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7419013Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7419212Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7419436Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7419781Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7420106Z     return mod(*inputs)
2025-09-07T09:17:41.7420472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7420871Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7421249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7421645Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7421986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7422338Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7422735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7423137Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7423541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7423959Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7424455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7424938Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7425117Z 
2025-09-07T09:17:41.7425212Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7425563Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7425876Z     return mod(*inputs)
2025-09-07T09:17:41.7426240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7426680Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7427058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7427455Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7427803Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7428156Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7428569Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7428977Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7429426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7429845Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7430291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7430779Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7430951Z 
2025-09-07T09:17:41.7431046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7431393Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7431706Z     return mod(*inputs)
2025-09-07T09:17:41.7432063Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7432460Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7432847Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7433232Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7433564Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7433937Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7434323Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7434725Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7435125Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7435527Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7435655Z 
2025-09-07T09:17:41.7435749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7436089Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7436410Z     return mod(*inputs)
2025-09-07T09:17:41.7436769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7437168Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7437551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7437949Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7438356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7438714Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7439100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7439556Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7439730Z 
2025-09-07T09:17:41.7439825Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7440171Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7440523Z     return mod(*inputs)
2025-09-07T09:17:41.7440884Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7441292Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7441674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7442064Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7442400Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7442744Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7443173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7443616Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7444004Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7444353Z     return self.act(input)
2025-09-07T09:17:41.7444458Z 
2025-09-07T09:17:41.7444582Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7444929Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7445248Z     return mod(*inputs)
2025-09-07T09:17:41.7445614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7446008Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7446386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7446784Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7447119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7447480Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7447862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7448261Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7448407Z 
2025-09-07T09:17:41.7448498Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7448836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7449153Z     return mod(*inputs)
2025-09-07T09:17:41.7449508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7449902Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7450291Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7450681Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7451016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7451366Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7451757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7452206Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7452617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7453036Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7453204Z 
2025-09-07T09:17:41.7453297Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7453639Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7453955Z     return mod(*inputs)
2025-09-07T09:17:41.7454347Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7454740Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7455114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7455510Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7455854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7456225Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7456620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7457068Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7457489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7458025Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7458248Z 
2025-09-07T09:17:41.7458392Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7458742Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7459063Z     return mod(*inputs)
2025-09-07T09:17:41.7459434Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7459833Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7460218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7460614Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7460955Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7461332Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7461726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7462141Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7462543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7463036Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7463256Z 
2025-09-07T09:17:41.7463339Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7463545Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7463732Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7463923Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7464145Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7464493Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7464807Z     return mod(*inputs)
2025-09-07T09:17:41.7465169Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7465614Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7466000Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7466402Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7466733Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7467087Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7467484Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7467886Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7468314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7468738Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7469189Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7469669Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7469851Z 
2025-09-07T09:17:41.7469955Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7470298Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7470621Z     return mod(*inputs)
2025-09-07T09:17:41.7471016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7471424Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7471806Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7472225Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7472562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7472919Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7473309Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7473718Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7474109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7474525Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7474965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7475423Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7475580Z 
2025-09-07T09:17:41.7475674Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7476017Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7476331Z     return mod(*inputs)
2025-09-07T09:17:41.7476691Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7477089Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7477472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7477863Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7478196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7478551Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7478941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7479381Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7479790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7480186Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7480319Z 
2025-09-07T09:17:41.7480421Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7480761Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7481078Z     return mod(*inputs)
2025-09-07T09:17:41.7481436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7481869Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7482252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7482635Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7482972Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7483327Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7483717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7484154Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7484322Z 
2025-09-07T09:17:41.7484449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7484792Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7485105Z     return mod(*inputs)
2025-09-07T09:17:41.7485463Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7485900Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7486276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7486663Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7486998Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7487350Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7487731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7488161Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7488539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7488878Z     return self.act(input)
2025-09-07T09:17:41.7488982Z 
2025-09-07T09:17:41.7489084Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7489416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7489726Z     return mod(*inputs)
2025-09-07T09:17:41.7490086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7490478Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7490850Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7491238Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7491579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7491931Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7492319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7492712Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7492891Z 
2025-09-07T09:17:41.7492982Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7493322Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7493636Z     return mod(*inputs)
2025-09-07T09:17:41.7493985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7494380Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7494760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7495150Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7495528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7495871Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7496254Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7496663Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7497066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7497492Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7497716Z 
2025-09-07T09:17:41.7497809Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7498187Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7498501Z     return mod(*inputs)
2025-09-07T09:17:41.7498857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7499255Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7499661Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7500058Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7500391Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7500749Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7501131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7501536Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7501936Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7502415Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7502630Z 
2025-09-07T09:17:41.7502730Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7503248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7503564Z     return mod(*inputs)
2025-09-07T09:17:41.7503926Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7504322Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7504701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7505085Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7505426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7505783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7506171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7506567Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7507033Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7507516Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7507745Z 
2025-09-07T09:17:41.7507820Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7508023Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7508210Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7508405Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7508622Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7508968Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7509332Z     return mod(*inputs)
2025-09-07T09:17:41.7509708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7510104Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7510493Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7510889Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7511221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7511577Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7512025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7512446Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7512851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7513309Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7513765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7514247Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7514428Z 
2025-09-07T09:17:41.7514534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7514882Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7515189Z     return mod(*inputs)
2025-09-07T09:17:41.7515554Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7515952Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7516345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7516763Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7517102Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7517460Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7517851Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7518256Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7518649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7519062Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7519499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7519954Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7520110Z 
2025-09-07T09:17:41.7520213Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7520549Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7520902Z     return mod(*inputs)
2025-09-07T09:17:41.7521268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7521665Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7522039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7522434Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7522771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7523230Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7523621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7524019Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7524419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7524825Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7524954Z 
2025-09-07T09:17:41.7525056Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7525394Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7525698Z     return mod(*inputs)
2025-09-07T09:17:41.7526090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7526488Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7526869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7527282Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7527623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7527972Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7528358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7528797Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7528961Z 
2025-09-07T09:17:41.7529052Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7529388Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7529702Z     return mod(*inputs)
2025-09-07T09:17:41.7530061Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7530452Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7530821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7531220Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7531560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7531917Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7532301Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7532733Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7533112Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7533456Z     return self.act(input)
2025-09-07T09:17:41.7533557Z 
2025-09-07T09:17:41.7533661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7534000Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7534360Z     return mod(*inputs)
2025-09-07T09:17:41.7534722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7535113Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7535497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7535879Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7536213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7536566Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7536990Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7537386Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7537599Z 
2025-09-07T09:17:41.7537693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7538035Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7538343Z     return mod(*inputs)
2025-09-07T09:17:41.7538700Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7539086Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7539523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7539919Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7540258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7540613Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7541035Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7541450Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7541857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7542285Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7542446Z 
2025-09-07T09:17:41.7542905Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7543242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7543555Z     return mod(*inputs)
2025-09-07T09:17:41.7543920Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7544314Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7544687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7545080Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7545414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7545769Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7546158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7546557Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7546967Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7547451Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7547671Z 
2025-09-07T09:17:41.7547776Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7548121Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7548479Z     return mod(*inputs)
2025-09-07T09:17:41.7548842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7549236Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7549614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7549997Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7550336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7550690Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7551115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7551521Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7551918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7552404Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7552629Z 
2025-09-07T09:17:41.7552703Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7552906Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7553101Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7553288Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7553535Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7553880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7554197Z     return mod(*inputs)
2025-09-07T09:17:41.7554583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7554984Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7555364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7555754Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7556103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7556468Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7556873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7557291Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7557702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7558124Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7558576Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7559067Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7559248Z 
2025-09-07T09:17:41.7559358Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7559714Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7560030Z     return mod(*inputs)
2025-09-07T09:17:41.7560409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7560814Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7561200Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7561592Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7561933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7562333Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7577845Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7578309Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7578731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7579161Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7579607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7581419Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7581579Z 
2025-09-07T09:17:41.7581686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7582035Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7582352Z     return mod(*inputs)
2025-09-07T09:17:41.7582723Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7583119Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7583500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7583889Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7584272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7584634Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7585054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7585462Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7585879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7586271Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7586398Z 
2025-09-07T09:17:41.7586496Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7586836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7587137Z     return mod(*inputs)
2025-09-07T09:17:41.7587492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7587876Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7588247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7588627Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7588956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7589303Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7589682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7590067Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7590190Z 
2025-09-07T09:17:41.7590279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7590612Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7590913Z     return mod(*inputs)
2025-09-07T09:17:41.7591259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7591640Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7592010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7592435Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7592757Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7593096Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7593472Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7593900Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7594066Z 
2025-09-07T09:17:41.7594155Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7594483Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7594826Z     return mod(*inputs)
2025-09-07T09:17:41.7595171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7595563Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7595928Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7596311Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7596632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7596968Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7597380Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7597804Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7598173Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7598531Z     return self.act(input)
2025-09-07T09:17:41.7598637Z 
2025-09-07T09:17:41.7598728Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7599057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7599360Z     return mod(*inputs)
2025-09-07T09:17:41.7599709Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7600091Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7600487Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7600875Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7601202Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7601547Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7601925Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7602316Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7602445Z 
2025-09-07T09:17:41.7602540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7602879Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7603371Z     return mod(*inputs)
2025-09-07T09:17:41.7603745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7604160Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7604530Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7604916Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7605240Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7605582Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7606070Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7606470Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7606861Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7607282Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7607442Z 
2025-09-07T09:17:41.7607534Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7607866Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7608227Z     return mod(*inputs)
2025-09-07T09:17:41.7608579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7608964Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7609337Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7609725Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7610048Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7610388Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7610812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7611216Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7611609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7612135Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7612354Z 
2025-09-07T09:17:41.7612447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7612780Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7613081Z     return mod(*inputs)
2025-09-07T09:17:41.7613428Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7613808Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7614178Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7614555Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7614879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7615221Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7615594Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7615989Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7616377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7616849Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7617066Z 
2025-09-07T09:17:41.7617140Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7617331Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7617513Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7617758Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7617961Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7618293Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7618596Z     return mod(*inputs)
2025-09-07T09:17:41.7618941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7619371Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7619742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7620119Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7620445Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7620788Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7621165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7621601Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7621994Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7622400Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7622834Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7623302Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7623482Z 
2025-09-07T09:17:41.7623571Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7623904Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7624241Z     return mod(*inputs)
2025-09-07T09:17:41.7624593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7624979Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7625377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7625762Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7626085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7626427Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7626800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7627196Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7627591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7627989Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7628417Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7628864Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7629022Z 
2025-09-07T09:17:41.7629115Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7629450Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7629753Z     return mod(*inputs)
2025-09-07T09:17:41.7630105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7630488Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7630859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7631240Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7631563Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7631904Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7632277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7632706Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7633094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7633482Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7633605Z 
2025-09-07T09:17:41.7633694Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7634021Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7634322Z     return mod(*inputs)
2025-09-07T09:17:41.7634666Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7635085Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7635448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7635828Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7636146Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7636484Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7636858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7637280Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7637443Z 
2025-09-07T09:17:41.7637560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7637886Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7638187Z     return mod(*inputs)
2025-09-07T09:17:41.7638562Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7638948Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7639316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7639693Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7640016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7640362Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7640737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7641153Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7641516Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7641841Z     return self.act(input)
2025-09-07T09:17:41.7641941Z 
2025-09-07T09:17:41.7642029Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7642357Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7642656Z     return mod(*inputs)
2025-09-07T09:17:41.7643001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7643380Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7643746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7644123Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7644449Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7644792Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7645179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7645616Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7645746Z 
2025-09-07T09:17:41.7645832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7646158Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7646456Z     return mod(*inputs)
2025-09-07T09:17:41.7646800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7647188Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7647559Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7647977Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7648299Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7648645Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7649023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7649423Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7649815Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7650237Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7650395Z 
2025-09-07T09:17:41.7650518Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7650855Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7651164Z     return mod(*inputs)
2025-09-07T09:17:41.7651510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7651951Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7652328Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7652716Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7653051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7653390Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7653774Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7654186Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7654581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7655058Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7655276Z 
2025-09-07T09:17:41.7655375Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7655717Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7656019Z     return mod(*inputs)
2025-09-07T09:17:41.7656388Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7656782Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7657155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7657604Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7657943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7658292Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7658670Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7659130Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7659528Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7660008Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7660232Z 
2025-09-07T09:17:41.7660312Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7660503Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7660688Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7660885Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7661097Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7661483Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7661792Z     return mod(*inputs)
2025-09-07T09:17:41.7662148Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7662534Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7662914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7663288Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7663622Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7663974Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7664386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7664793Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7665212Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7665623Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7666058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7666525Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7666700Z 
2025-09-07T09:17:41.7666794Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7667132Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7667435Z     return mod(*inputs)
2025-09-07T09:17:41.7667791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7668183Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7668551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7668941Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7669266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7669645Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7670023Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7670418Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7670811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7671223Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7671655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7672110Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7672263Z 
2025-09-07T09:17:41.7672351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7672734Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7673052Z     return mod(*inputs)
2025-09-07T09:17:41.7673406Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7673789Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7674165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7674551Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7674877Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7675265Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7675646Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7676043Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7676433Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7676828Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7676952Z 
2025-09-07T09:17:41.7677046Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7677371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7677715Z     return mod(*inputs)
2025-09-07T09:17:41.7678069Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7678459Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7678858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7679253Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7679586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7679934Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7680315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7680703Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7680828Z 
2025-09-07T09:17:41.7680919Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7681249Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7681549Z     return mod(*inputs)
2025-09-07T09:17:41.7681899Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7682286Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7682657Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7683034Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7683353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7683702Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7684079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7684508Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7684669Z 
2025-09-07T09:17:41.7684762Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7685095Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7685396Z     return mod(*inputs)
2025-09-07T09:17:41.7685746Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7686208Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7686581Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7686965Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7687288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7687632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7688011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7688474Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7688839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7689169Z     return self.act(input)
2025-09-07T09:17:41.7689275Z 
2025-09-07T09:17:41.7689364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7689695Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7689993Z     return mod(*inputs)
2025-09-07T09:17:41.7690336Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7690716Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7691120Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7691502Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7691822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7692199Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7692574Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7692971Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7693102Z 
2025-09-07T09:17:41.7693197Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7693523Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7693826Z     return mod(*inputs)
2025-09-07T09:17:41.7694174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7694560Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7694934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7695313Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7695639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7695983Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7696361Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7696753Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7697147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7697628Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7697787Z 
2025-09-07T09:17:41.7697882Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7698210Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7698508Z     return mod(*inputs)
2025-09-07T09:17:41.7698859Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7699288Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7699660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7700041Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7700359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7700697Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7701078Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7701472Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7701894Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7702371Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7702584Z 
2025-09-07T09:17:41.7702672Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7703006Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7703444Z     return mod(*inputs)
2025-09-07T09:17:41.7703811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7704208Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7704679Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7705069Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7705399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7705790Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7706179Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7706599Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7706995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7707474Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7707707Z 
2025-09-07T09:17:41.7707781Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7707972Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7708153Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7708334Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7708540Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7708879Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7709185Z     return mod(*inputs)
2025-09-07T09:17:41.7709542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7709943Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7710315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7710703Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7711041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7711392Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7711772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7712179Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7712577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7713055Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7713503Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7713979Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7714166Z 
2025-09-07T09:17:41.7714261Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7714606Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7714916Z     return mod(*inputs)
2025-09-07T09:17:41.7715277Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7715725Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7716106Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7716502Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7716844Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7717191Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7717577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7717979Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7718414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7718832Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7719295Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7719755Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7719921Z 
2025-09-07T09:17:41.7720011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7720353Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7720661Z     return mod(*inputs)
2025-09-07T09:17:41.7721012Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7721403Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7721781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7722179Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7722505Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7722861Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7723247Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7723654Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7724055Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7724441Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7724576Z 
2025-09-07T09:17:41.7724668Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7725000Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7725309Z     return mod(*inputs)
2025-09-07T09:17:41.7725660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7726045Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7726419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7726847Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7727174Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7727520Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7727897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7728334Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7728504Z 
2025-09-07T09:17:41.7728598Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7728969Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7729270Z     return mod(*inputs)
2025-09-07T09:17:41.7729634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7730025Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7730402Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7730790Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7731116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7731466Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7731885Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7732322Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7732736Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7733076Z     return self.act(input)
2025-09-07T09:17:41.7733184Z 
2025-09-07T09:17:41.7733275Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7733613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7733932Z     return mod(*inputs)
2025-09-07T09:17:41.7734280Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7734668Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7735044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7735425Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7735753Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7736092Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7736476Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7736878Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7737010Z 
2025-09-07T09:17:41.7737105Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7737439Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7737827Z     return mod(*inputs)
2025-09-07T09:17:41.7738199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7738588Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7738963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7739347Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7739673Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7740087Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7740469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7740881Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7741274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7741691Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7741870Z 
2025-09-07T09:17:41.7741968Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7742306Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7742654Z     return mod(*inputs)
2025-09-07T09:17:41.7743005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7743391Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7743786Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7744179Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7744508Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7744851Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7745272Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7745677Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7746082Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7746622Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7746845Z 
2025-09-07T09:17:41.7746937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7747274Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7747581Z     return mod(*inputs)
2025-09-07T09:17:41.7747947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7748328Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7748697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7749078Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7749414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7749756Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7750155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7750556Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7750943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7751424Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7751643Z 
2025-09-07T09:17:41.7751715Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7751919Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7752105Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7752294Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7752504Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7752836Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7753157Z     return mod(*inputs)
2025-09-07T09:17:41.7753548Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7753931Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7754297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7754681Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7755005Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7755347Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7755728Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7756186Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7756578Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7757002Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7757443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7757914Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7758097Z 
2025-09-07T09:17:41.7758194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7758597Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7758915Z     return mod(*inputs)
2025-09-07T09:17:41.7759274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7759665Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7760090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7760499Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7760849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7761203Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7761588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7761994Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7762394Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7762799Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7763234Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7763698Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7763859Z 
2025-09-07T09:17:41.7763963Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7764309Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7764617Z     return mod(*inputs)
2025-09-07T09:17:41.7764977Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7765368Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7765741Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7766133Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7766465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7766810Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7767196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7767636Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7768034Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7768429Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7768555Z 
2025-09-07T09:17:41.7768647Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7768983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7769291Z     return mod(*inputs)
2025-09-07T09:17:41.7769694Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7770090Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7770459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7770844Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7771171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7771521Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7771896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7772333Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7772461Z 
2025-09-07T09:17:41.7772552Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7772897Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7773203Z     return mod(*inputs)
2025-09-07T09:17:41.7773586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7773986Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7774386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7774768Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7775094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7775453Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7775840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7776267Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7776431Z 
2025-09-07T09:17:41.7776525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7776865Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7777178Z     return mod(*inputs)
2025-09-07T09:17:41.7777593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7777992Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7778251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7778324Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7778545Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7778615Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7778881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7778987Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7779205Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7779309Z     return self.act(input)
2025-09-07T09:17:41.7779313Z 
2025-09-07T09:17:41.7779405Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7779597Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7779654Z     return mod(*inputs)
2025-09-07T09:17:41.7779924Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7779991Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7780259Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7780362Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7780580Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7780658Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7780919Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7781000Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7781004Z 
2025-09-07T09:17:41.7781094Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7781276Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7781396Z     return mod(*inputs)
2025-09-07T09:17:41.7781658Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7781734Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7782050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7782115Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7782339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7782410Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7782674Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7782758Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7783025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7783129Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7783134Z 
2025-09-07T09:17:41.7783224Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7783416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7783472Z     return mod(*inputs)
2025-09-07T09:17:41.7783740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7783805Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7784064Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7784130Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7784343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7784423Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7784682Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7784769Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7785026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7785222Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7785225Z 
2025-09-07T09:17:41.7785324Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7785508Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7785573Z     return mod(*inputs)
2025-09-07T09:17:41.7785837Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7785903Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7786165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7786262Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7786485Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7786556Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7786822Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7786901Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7787161Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7787356Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7787360Z 
2025-09-07T09:17:41.7787435Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7787513Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7787583Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7787656Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7787787Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7787982Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7788048Z     return mod(*inputs)
2025-09-07T09:17:41.7788327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7788393Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7788663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7788728Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7788953Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7789024Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7789289Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7789373Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7789633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7789730Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7790026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7790156Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7790159Z 
2025-09-07T09:17:41.7790252Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7790440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7790505Z     return mod(*inputs)
2025-09-07T09:17:41.7790769Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7790840Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7791147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7791220Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7791436Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7791503Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7791776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7791858Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7792124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7792241Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7792535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7792643Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7792646Z 
2025-09-07T09:17:41.7792736Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7792927Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7792986Z     return mod(*inputs)
2025-09-07T09:17:41.7793288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7793354Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7793614Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7793687Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7793931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7794009Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7794268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7794347Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7794611Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7794684Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7794689Z 
2025-09-07T09:17:41.7794786Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7794971Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7795029Z     return mod(*inputs)
2025-09-07T09:17:41.7795297Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7795365Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7795627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7795689Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7795909Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7795976Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7796236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7796350Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7796355Z 
2025-09-07T09:17:41.7796446Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7796639Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7796697Z     return mod(*inputs)
2025-09-07T09:17:41.7797001Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7797073Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7797330Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7797401Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7797616Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7797688Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7797948Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7798085Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7798302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7798365Z     return self.act(input)
2025-09-07T09:17:41.7798369Z 
2025-09-07T09:17:41.7798466Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7798651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7798709Z     return mod(*inputs)
2025-09-07T09:17:41.7798976Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7799072Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7799341Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7799410Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7799668Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7799738Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7799997Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7800079Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7800082Z 
2025-09-07T09:17:41.7800173Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7800364Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7800421Z     return mod(*inputs)
2025-09-07T09:17:41.7800685Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7800758Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7801015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7801087Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7801300Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7801368Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7801633Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7801714Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7801985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7802086Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7802090Z 
2025-09-07T09:17:41.7802188Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7802371Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7802432Z     return mod(*inputs)
2025-09-07T09:17:41.7802698Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7802796Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7803211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7803277Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7803491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7803569Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7803829Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7803987Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7804248Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7804413Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7804417Z 
2025-09-07T09:17:41.7804507Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7804690Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7804755Z     return mod(*inputs)
2025-09-07T09:17:41.7805016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7805134Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7805395Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7805463Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7805727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7805799Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7806067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7806148Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7806413Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7809092Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7809103Z 
2025-09-07T09:17:41.7809200Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7809279Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7809349Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7809418Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7809531Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7809740Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7809809Z     return mod(*inputs)
2025-09-07T09:17:41.7810097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7810171Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7810450Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7810517Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7810785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7810862Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7811124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7811222Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7811483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7811651Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7811952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7812081Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7812089Z 
2025-09-07T09:17:41.7812194Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7812392Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7812493Z     return mod(*inputs)
2025-09-07T09:17:41.7812759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7812835Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7813097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7813165Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7813386Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7813461Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7813727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7813855Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7814117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7814219Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7814514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7814626Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7814629Z 
2025-09-07T09:17:41.7814726Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7814922Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7814981Z     return mod(*inputs)
2025-09-07T09:17:41.7815319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7815399Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7815659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7815733Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7815949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7816021Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7816286Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7816368Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7816629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7816701Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7816706Z 
2025-09-07T09:17:41.7816805Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7816991Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7817050Z     return mod(*inputs)
2025-09-07T09:17:41.7817319Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7817383Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7817743Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7817807Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7818022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7818097Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7818358Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7818437Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7818440Z 
2025-09-07T09:17:41.7818570Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7818763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7818821Z     return mod(*inputs)
2025-09-07T09:17:41.7819083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7819156Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7819415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7819484Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7819697Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7819799Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7820066Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7820182Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7820185Z 
2025-09-07T09:17:41.7820285Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7820471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7820529Z     return mod(*inputs)
2025-09-07T09:17:41.7820799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7820865Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7821129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7821244Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7821469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7821543Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7821808Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7821919Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7822128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7822191Z     return self.act(input)
2025-09-07T09:17:41.7822195Z 
2025-09-07T09:17:41.7822291Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7822481Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7822545Z     return mod(*inputs)
2025-09-07T09:17:41.7822813Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7822887Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7823149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7823221Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7823439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7823548Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7823819Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7823898Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7823902Z 
2025-09-07T09:17:41.7824006Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7824201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7824264Z     return mod(*inputs)
2025-09-07T09:17:41.7824540Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7824643Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7824911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7824977Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7825196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7825277Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7825539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7825630Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7825918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7826031Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7826035Z 
2025-09-07T09:17:41.7826127Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7826315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7826382Z     return mod(*inputs)
2025-09-07T09:17:41.7826644Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7826714Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7826974Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7827038Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7827298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7827371Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7827640Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7827721Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7827985Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7828144Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7828147Z 
2025-09-07T09:17:41.7828236Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7828427Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7828484Z     return mod(*inputs)
2025-09-07T09:17:41.7828754Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7828819Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7829079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7829150Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7829363Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7829474Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7829735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7829819Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7830079Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7830238Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7830242Z 
2025-09-07T09:17:41.7830355Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7830425Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7830503Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7830571Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7830662Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7830855Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7830912Z     return mod(*inputs)
2025-09-07T09:17:41.7831183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7831246Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7831537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7831611Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7831825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7831905Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7832160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7832250Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7832511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7832603Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7832906Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7833068Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7833072Z 
2025-09-07T09:17:41.7833171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7833359Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7833418Z     return mod(*inputs)
2025-09-07T09:17:41.7833689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7833761Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7834029Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7834094Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7834314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7834385Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7834649Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7834735Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7834993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7835082Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7835409Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7835513Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7835517Z 
2025-09-07T09:17:41.7835614Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7835800Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7835866Z     return mod(*inputs)
2025-09-07T09:17:41.7836129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7836204Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7836522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7836584Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7841940Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7842056Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7842371Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7842463Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7842832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7842919Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7842923Z 
2025-09-07T09:17:41.7843027Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7843242Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7843302Z     return mod(*inputs)
2025-09-07T09:17:41.7843591Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7843668Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7843931Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7843998Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7844218Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7844336Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7844607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7844723Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7844727Z 
2025-09-07T09:17:41.7844832Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7845032Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7845093Z     return mod(*inputs)
2025-09-07T09:17:41.7845356Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7845420Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7845683Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7845748Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7845973Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7846043Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7846302Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7846407Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7846660Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7846726Z     return self.act(input)
2025-09-07T09:17:41.7846729Z 
2025-09-07T09:17:41.7846823Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7847019Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7847078Z     return mod(*inputs)
2025-09-07T09:17:41.7847343Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7847409Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7847704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7847773Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7847987Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7848055Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7848315Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7848391Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7848395Z 
2025-09-07T09:17:41.7848491Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7848949Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7849011Z     return mod(*inputs)
2025-09-07T09:17:41.7849274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7849340Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7849597Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7849661Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7849876Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7849942Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7850197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7850285Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7850586Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7850695Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7850700Z 
2025-09-07T09:17:41.7850793Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7850983Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7851040Z     return mod(*inputs)
2025-09-07T09:17:41.7851298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7851363Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7851620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7851684Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7851900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7851965Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7852224Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7852303Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7852561Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7852755Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7852759Z 
2025-09-07T09:17:41.7852852Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7853038Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7853095Z     return mod(*inputs)
2025-09-07T09:17:41.7853360Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7853420Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7853718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7853781Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7853992Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7854066Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7854320Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7854399Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7854653Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7854847Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7854851Z 
2025-09-07T09:17:41.7854927Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7854999Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7855071Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7855138Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7855234Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7855421Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7855476Z     return mod(*inputs)
2025-09-07T09:17:41.7855740Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7855804Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7856105Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7856173Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7856385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7856459Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7856715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7856798Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7857054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7857144Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7857443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7857642Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7857647Z 
2025-09-07T09:17:41.7857740Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7857925Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7857985Z     return mod(*inputs)
2025-09-07T09:17:41.7858246Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7858344Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7858609Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7858674Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7858895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7858965Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7859227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7859315Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7859610Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7859702Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7859993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7860103Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7860107Z 
2025-09-07T09:17:41.7860202Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7860394Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7860452Z     return mod(*inputs)
2025-09-07T09:17:41.7860752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7860822Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7861086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7861149Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7861369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7861441Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7861702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7861784Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7862044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7862154Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7862157Z 
2025-09-07T09:17:41.7862249Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7862438Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7862496Z     return mod(*inputs)
2025-09-07T09:17:41.7862759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7862823Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7863080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7863143Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7863357Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7863430Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7863689Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7863763Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7863766Z 
2025-09-07T09:17:41.7863853Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7864041Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7864155Z     return mod(*inputs)
2025-09-07T09:17:41.7864416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7864483Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7864738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7864800Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7865022Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7865089Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7865423Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7865534Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7865537Z 
2025-09-07T09:17:41.7865633Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7865820Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7865877Z     return mod(*inputs)
2025-09-07T09:17:41.7866144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7866205Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7866502Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7866565Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7866780Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7866857Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7867110Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7867218Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7867426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7867489Z     return self.act(input)
2025-09-07T09:17:41.7867497Z 
2025-09-07T09:17:41.7867589Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7867808Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7867869Z     return mod(*inputs)
2025-09-07T09:17:41.7868129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7868199Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7868454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7868515Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7868731Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7868796Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7869053Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7869124Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7869128Z 
2025-09-07T09:17:41.7869219Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7869407Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7869464Z     return mod(*inputs)
2025-09-07T09:17:41.7869726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7869789Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7870083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7870151Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7870364Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7870437Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7870693Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7870776Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7871032Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7871170Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7871174Z 
2025-09-07T09:17:41.7871268Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7871451Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7871509Z     return mod(*inputs)
2025-09-07T09:17:41.7871768Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7871833Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7872097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7872190Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7872407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7872477Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7872738Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7872817Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7873072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7873228Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7873232Z 
2025-09-07T09:17:41.7873320Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7873540Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7873599Z     return mod(*inputs)
2025-09-07T09:17:41.7873862Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7873931Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7874187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7874253Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7874465Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7874534Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7874789Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7874864Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7875126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7875282Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7875286Z 
2025-09-07T09:17:41.7875361Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7875429Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7875496Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7875603Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7875689Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7875877Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7875932Z     return mod(*inputs)
2025-09-07T09:17:41.7876192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7876258Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7876519Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7876619Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7876830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7876901Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7877163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7877239Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7877501Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7877587Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7877918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7878042Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7878047Z 
2025-09-07T09:17:41.7878134Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7878320Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7878375Z     return mod(*inputs)
2025-09-07T09:17:41.7878639Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7878703Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7878963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7879024Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7879268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7879339Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7879598Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7879687Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7879941Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7880027Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7880317Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7880414Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7880417Z 
2025-09-07T09:17:41.7880509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7880696Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7880757Z     return mod(*inputs)
2025-09-07T09:17:41.7881025Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7881090Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7881350Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7881449Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7881663Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7881728Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7881986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7882068Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7882327Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7882400Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7882435Z 
2025-09-07T09:17:41.7882525Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7882712Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7882768Z     return mod(*inputs)
2025-09-07T09:17:41.7883028Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7883094Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7883351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7883418Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7883664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7883735Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7883996Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7884103Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7884107Z 
2025-09-07T09:17:41.7884198Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7884383Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7884439Z     return mod(*inputs)
2025-09-07T09:17:41.7884702Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7884764Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7885075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7885138Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7885353Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7885420Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7885678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7885786Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7885995Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7886057Z     return self.act(input)
2025-09-07T09:17:41.7886060Z 
2025-09-07T09:17:41.7886152Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7886333Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7886394Z     return mod(*inputs)
2025-09-07T09:17:41.7886652Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7886719Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7886975Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7887038Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7887288Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7887354Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7887617Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7887688Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7887692Z 
2025-09-07T09:17:41.7887784Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7887968Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7888024Z     return mod(*inputs)
2025-09-07T09:17:41.7888318Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7888382Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7888643Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7888706Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7888918Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7888991Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7889244Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7889356Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7889613Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7889719Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7889722Z 
2025-09-07T09:17:41.7889812Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7889994Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7890055Z     return mod(*inputs)
2025-09-07T09:17:41.7890316Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7890381Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7890637Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7890736Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7890964Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7891036Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7891296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7891378Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7891636Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7891787Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7891791Z 
2025-09-07T09:17:41.7891885Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7892078Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7892137Z     return mod(*inputs)
2025-09-07T09:17:41.7892404Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7892470Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7892727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7892791Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7893041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7893114Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7893369Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7893449Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7893708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7893862Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7893897Z 
2025-09-07T09:17:41.7893976Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7894043Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7894114Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7894182Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7894276Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7894464Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7894520Z     return mod(*inputs)
2025-09-07T09:17:41.7894785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7894847Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7895138Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7895206Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7895421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7895491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7895748Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7895834Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7896089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7896173Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7896513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7896638Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7896643Z 
2025-09-07T09:17:41.7896737Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7896923Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7896981Z     return mod(*inputs)
2025-09-07T09:17:41.7897242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7897307Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7897621Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7897683Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7897901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7897973Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7898230Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7898310Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7898566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7898651Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7898980Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7899084Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7899087Z 
2025-09-07T09:17:41.7899179Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7899366Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7899433Z     return mod(*inputs)
2025-09-07T09:17:41.7899692Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7899801Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7900058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7900124Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7900340Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7900407Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7900669Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7900746Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7901044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7901115Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7901120Z 
2025-09-07T09:17:41.7901214Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7901399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7901455Z     return mod(*inputs)
2025-09-07T09:17:41.7901722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7901788Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7902044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7902110Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7902359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7902429Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7902687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7902763Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7902767Z 
2025-09-07T09:17:41.7902856Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7903202Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7903265Z     return mod(*inputs)
2025-09-07T09:17:41.7903525Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7903592Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7903849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7903914Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7904128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7904197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7904457Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7904569Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7904651Z 
2025-09-07T09:17:41.7904748Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7904928Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7904984Z     return mod(*inputs)
2025-09-07T09:17:41.7905249Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7905311Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7905572Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7905687Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7905900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7905969Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7906227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7906334Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7906542Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7906603Z     return self.act(input)
2025-09-07T09:17:41.7906608Z 
2025-09-07T09:17:41.7906696Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7906943Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7907004Z     return mod(*inputs)
2025-09-07T09:17:41.7907268Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7907334Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7907592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7907658Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7907870Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7907939Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7908203Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7908331Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7908335Z 
2025-09-07T09:17:41.7908427Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7908619Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7908679Z     return mod(*inputs)
2025-09-07T09:17:41.7908943Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7909006Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7909265Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7909327Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7909537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7909608Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7909869Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7909950Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7910209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7910311Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7910350Z 
2025-09-07T09:17:41.7910447Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7910630Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7910689Z     return mod(*inputs)
2025-09-07T09:17:41.7910949Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7911012Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7911276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7911336Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7911592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7911660Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7911922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7912002Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7912260Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7912415Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7912418Z 
2025-09-07T09:17:41.7912509Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7912727Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7912787Z     return mod(*inputs)
2025-09-07T09:17:41.7913051Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7913119Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7913374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7913445Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7913655Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7913727Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7913982Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7914106Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7914374Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7914531Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7914535Z 
2025-09-07T09:17:41.7914609Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7914678Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7914749Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7914821Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7914910Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7915096Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7915151Z     return mod(*inputs)
2025-09-07T09:17:41.7915412Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7915482Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7915737Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7915802Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7916015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7916121Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7916376Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7916451Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7916713Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7916800Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7917100Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7917252Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7917255Z 
2025-09-07T09:17:41.7917344Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7917532Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7917590Z     return mod(*inputs)
2025-09-07T09:17:41.7917855Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7917918Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7918181Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7918244Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7918489Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7918562Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7918821Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7918901Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7919155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7919242Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7919535Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7919635Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7919639Z 
2025-09-07T09:17:41.7919767Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7919953Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7920016Z     return mod(*inputs)
2025-09-07T09:17:41.7920276Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7920339Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7920600Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7920665Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7920879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7920943Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7921199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7921282Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7921543Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7921615Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7921619Z 
2025-09-07T09:17:41.7921706Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7921895Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7921996Z     return mod(*inputs)
2025-09-07T09:17:41.7922252Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7922317Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7922573Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7922643Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7922857Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7922958Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7923221Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7923330Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7923336Z 
2025-09-07T09:17:41.7923429Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7923608Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7923665Z     return mod(*inputs)
2025-09-07T09:17:41.7923927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7923988Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7924298Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7924360Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7924577Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7924643Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7924903Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7925006Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7925211Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7925274Z     return self.act(input)
2025-09-07T09:17:41.7925278Z 
2025-09-07T09:17:41.7925367Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7925592Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7925651Z     return mod(*inputs)
2025-09-07T09:17:41.7925911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7925982Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7926236Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7926304Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7926515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7926581Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7926840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7926913Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7926918Z 
2025-09-07T09:17:41.7927011Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7927192Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7927250Z     return mod(*inputs)
2025-09-07T09:17:41.7927514Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7927615Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7927871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7927934Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7928147Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7928219Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7928475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7928555Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7928868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7928972Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7928975Z 
2025-09-07T09:17:41.7929068Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7929248Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7929308Z     return mod(*inputs)
2025-09-07T09:17:41.7929566Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7929629Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7929917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7929982Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7930196Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7930264Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7930522Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7930598Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7930858Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7931007Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7931010Z 
2025-09-07T09:17:41.7931097Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7931315Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7931371Z     return mod(*inputs)
2025-09-07T09:17:41.7931632Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7931697Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7931952Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7932019Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7932227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7932296Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7932550Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7932629Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7932886Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7933042Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7933046Z 
2025-09-07T09:17:41.7933119Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7933188Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7933298Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7933366Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7933453Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7933637Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7933691Z     return mod(*inputs)
2025-09-07T09:17:41.7933954Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7934020Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7934274Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7934379Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7934588Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7934656Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7934917Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7934996Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7935251Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7935335Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7935659Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7935777Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7935782Z 
2025-09-07T09:17:41.7935875Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7936057Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7936113Z     return mod(*inputs)
2025-09-07T09:17:41.7936377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7936439Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7936696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7936760Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7937006Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7937073Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7937331Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7937410Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7937708Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7937795Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7938081Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7938178Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7938184Z 
2025-09-07T09:17:41.7938272Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7938459Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7938520Z     return mod(*inputs)
2025-09-07T09:17:41.7938781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7938846Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7939099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7939198Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7939415Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7939479Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7939739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7939825Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7940094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7940210Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7940213Z 
2025-09-07T09:17:41.7940312Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7940508Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7940568Z     return mod(*inputs)
2025-09-07T09:17:41.7940836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7940903Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7941160Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7941226Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7941477Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7941550Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7941811Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7941881Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7941884Z 
2025-09-07T09:17:41.7941984Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7942172Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7942229Z     return mod(*inputs)
2025-09-07T09:17:41.7942492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7942557Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7942849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7942913Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7943129Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7943197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7943455Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7943568Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7943571Z 
2025-09-07T09:17:41.7943661Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7943846Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7943901Z     return mod(*inputs)
2025-09-07T09:17:41.7944166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7944228Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7944483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7944549Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7944760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7944862Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7945117Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7945220Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7945426Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7945488Z     return self.act(input)
2025-09-07T09:17:41.7945493Z 
2025-09-07T09:17:41.7945585Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7945768Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7945862Z     return mod(*inputs)
2025-09-07T09:17:41.7946121Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7946183Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7946443Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7946505Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7946717Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7946783Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7947072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7947150Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7947153Z 
2025-09-07T09:17:41.7947245Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7947433Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7947488Z     return mod(*inputs)
2025-09-07T09:17:41.7947749Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7947816Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7948072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7948138Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7948351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7948455Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7948714Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7948800Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7949058Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7949166Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7949169Z 
2025-09-07T09:17:41.7949264Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7949447Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7949506Z     return mod(*inputs)
2025-09-07T09:17:41.7949765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7949831Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7950091Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7950155Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7950368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7950433Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7950742Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7950827Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7951084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7951240Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7951246Z 
2025-09-07T09:17:41.7951339Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7951530Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7951620Z     return mod(*inputs)
2025-09-07T09:17:41.7951881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7951947Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7952204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7952271Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7952482Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7952549Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7952843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7952924Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7953184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7953358Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7953362Z 
2025-09-07T09:17:41.7953446Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7953514Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7953583Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7953654Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7953742Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7953928Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7953985Z     return mod(*inputs)
2025-09-07T09:17:41.7954275Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7954347Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7954606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7954674Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7954908Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7954980Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7955239Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7955315Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7955579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7955669Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7955965Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7956092Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7956096Z 
2025-09-07T09:17:41.7956184Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7956443Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7956499Z     return mod(*inputs)
2025-09-07T09:17:41.7956767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7956830Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7957089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7957156Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7957368Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7957471Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7957726Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7957803Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7958065Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7958150Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7958444Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7958547Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7958552Z 
2025-09-07T09:17:41.7958679Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7958867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7958925Z     return mod(*inputs)
2025-09-07T09:17:41.7959192Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7959258Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7959523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7959585Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7959799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7959874Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7960165Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7960249Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7960507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7960584Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7960587Z 
2025-09-07T09:17:41.7960677Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7960862Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7960922Z     return mod(*inputs)
2025-09-07T09:17:41.7961182Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7961250Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7961511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7961577Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7961795Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7961864Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7962126Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7962267Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7962270Z 
2025-09-07T09:17:41.7962364Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7962547Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7962604Z     return mod(*inputs)
2025-09-07T09:17:41.7962873Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7962937Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7963199Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7963295Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7963510Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7963583Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7963841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7963944Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7964151Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7964220Z     return self.act(input)
2025-09-07T09:17:41.7964223Z 
2025-09-07T09:17:41.7964347Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7964532Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7964596Z     return mod(*inputs)
2025-09-07T09:17:41.7964856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7964921Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7965186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7965248Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7965464Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7965530Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7965824Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7965897Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7965901Z 
2025-09-07T09:17:41.7965989Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7966175Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7966230Z     return mod(*inputs)
2025-09-07T09:17:41.7966494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7966558Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7966817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7966879Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7967089Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7967161Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7967418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7967503Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7967758Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7967855Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7967909Z 
2025-09-07T09:17:41.7967999Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7968184Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7968245Z     return mod(*inputs)
2025-09-07T09:17:41.7968504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7968572Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7968832Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7968929Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7969140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7969205Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7969466Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7969542Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7969796Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7969953Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7969957Z 
2025-09-07T09:17:41.7970079Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7970269Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7970326Z     return mod(*inputs)
2025-09-07T09:17:41.7970587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7970650Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7970912Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7970976Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7971186Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7971257Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7971560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7971636Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7971896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7972052Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7972055Z 
2025-09-07T09:17:41.7972132Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7972201Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7972273Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7972339Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7972428Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7972613Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7972670Z     return mod(*inputs)
2025-09-07T09:17:41.7972934Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7972996Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7973253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7973317Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7973527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7973632Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7973888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7973964Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7974222Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7974314Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7974606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7974760Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7974763Z 
2025-09-07T09:17:41.7974853Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7975037Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7975094Z     return mod(*inputs)
2025-09-07T09:17:41.7975355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7975418Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7975678Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7975773Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7975988Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7976058Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7976314Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7976394Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7976651Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7976734Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7977026Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7977119Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7977123Z 
2025-09-07T09:17:41.7977246Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7977433Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7977493Z     return mod(*inputs)
2025-09-07T09:17:41.7977792Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7977857Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7978124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7978187Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7978405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7978473Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7978735Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7978817Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7979075Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7979153Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7979156Z 
2025-09-07T09:17:41.7979246Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7979471Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7979530Z     return mod(*inputs)
2025-09-07T09:17:41.7979790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7979860Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7980116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7980186Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7980397Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7980499Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7980759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.7980832Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.7980836Z 
2025-09-07T09:17:41.7980933Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7981118Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7981180Z     return mod(*inputs)
2025-09-07T09:17:41.7981439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7981505Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7981799Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7981865Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7982084Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7982150Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7982408Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7982517Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7982520Z 
2025-09-07T09:17:41.7982611Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7982800Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7982857Z     return mod(*inputs)
2025-09-07T09:17:41.7983155Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7983229Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7983491Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7983558Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7983771Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7983845Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7984103Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.7984204Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.7984416Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.7984479Z     return self.act(input)
2025-09-07T09:17:41.7984482Z 
2025-09-07T09:17:41.7984576Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7984763Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7984821Z     return mod(*inputs)
2025-09-07T09:17:41.7985086Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7985183Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7985448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7985512Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7985727Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7985796Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7986054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.7986165Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.7986168Z 
2025-09-07T09:17:41.7986256Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7986445Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7986502Z     return mod(*inputs)
2025-09-07T09:17:41.7986763Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7986830Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7987087Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7987156Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7987403Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7987471Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7987739Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7987818Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7988077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.7988180Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.7988183Z 
2025-09-07T09:17:41.7988279Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7988463Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7988519Z     return mod(*inputs)
2025-09-07T09:17:41.7988823Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7988887Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7989152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7989215Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7989427Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7989503Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7989759Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7989839Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7990098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.7990258Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7990261Z 
2025-09-07T09:17:41.7990351Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7990534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7990594Z     return mod(*inputs)
2025-09-07T09:17:41.7990854Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7990957Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7991215Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7991277Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7991496Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7991565Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7991827Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7991938Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7992197Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.7992349Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.7992354Z 
2025-09-07T09:17:41.7992427Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7992502Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7992569Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7992644Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.7992733Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7992920Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7993031Z     return mod(*inputs)
2025-09-07T09:17:41.7993294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7993367Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7993627Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7993689Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7993911Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7993979Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7994238Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7994313Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7994607Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7994694Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7994986Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.7995113Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.7995118Z 
2025-09-07T09:17:41.7995207Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7995399Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7995454Z     return mod(*inputs)
2025-09-07T09:17:41.7995715Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7995785Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7996044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7996111Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7996326Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7996399Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7996656Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7996768Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7997031Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.7997114Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.7997407Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.7997508Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.7997511Z 
2025-09-07T09:17:41.7997600Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7997830Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7997889Z     return mod(*inputs)
2025-09-07T09:17:41.7998158Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.7998222Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.7998483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.7998547Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.7998761Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.7998835Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.7999124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.7999210Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.7999468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.7999540Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.7999545Z 
2025-09-07T09:17:41.7999639Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.7999821Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.7999884Z     return mod(*inputs)
2025-09-07T09:17:41.8000144Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8000213Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8000504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8000568Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8000785Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8000853Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8001119Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8001225Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8001228Z 
2025-09-07T09:17:41.8001317Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8001505Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8001562Z     return mod(*inputs)
2025-09-07T09:17:41.8001833Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8001896Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8002156Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8002225Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8002435Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8002541Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8002798Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8002904Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8003270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.8003335Z     return self.act(input)
2025-09-07T09:17:41.8003337Z 
2025-09-07T09:17:41.8003431Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8003616Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8003749Z     return mod(*inputs)
2025-09-07T09:17:41.8004010Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8004081Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8004345Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8004408Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8004623Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8004691Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8005002Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.8005077Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.8005082Z 
2025-09-07T09:17:41.8005171Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8005358Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8005417Z     return mod(*inputs)
2025-09-07T09:17:41.8005688Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8005752Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8006009Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8006076Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8006339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8006416Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8006672Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8006752Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8007015Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.8007115Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.8007118Z 
2025-09-07T09:17:41.8007211Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8007394Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8007456Z     return mod(*inputs)
2025-09-07T09:17:41.8007718Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8007779Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8008040Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8008105Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8008321Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8008438Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8008696Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8008778Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8009039Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.8009200Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8009204Z 
2025-09-07T09:17:41.8009293Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8009516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8009576Z     return mod(*inputs)
2025-09-07T09:17:41.8009839Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8009911Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8010168Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8010237Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8010447Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8010516Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8010812Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8010892Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8011157Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.8011312Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8011316Z 
2025-09-07T09:17:41.8011393Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8011460Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8011526Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8011596Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8011686Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8011874Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8011965Z     return mod(*inputs)
2025-09-07T09:17:41.8012227Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8012297Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8012558Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8012624Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8012836Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8012904Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8013166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8013242Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8013511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8013595Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8013896Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.8014012Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.8014016Z 
2025-09-07T09:17:41.8014107Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8014335Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8014393Z     return mod(*inputs)
2025-09-07T09:17:41.8014662Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8014723Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8014981Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8015047Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8015258Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8015377Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8015634Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8015722Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8015978Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8016061Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8016355Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.8016481Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.8016485Z 
2025-09-07T09:17:41.8016580Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8016766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8016825Z     return mod(*inputs)
2025-09-07T09:17:41.8017096Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8017161Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8017419Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8017482Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8017751Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8017822Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8018115Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8018197Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8018454Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.8018532Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.8018537Z 
2025-09-07T09:17:41.8018624Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8018806Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8018866Z     return mod(*inputs)
2025-09-07T09:17:41.8019131Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8019202Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8019461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8019532Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8019745Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8019809Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8020076Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.8020181Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.8020184Z 
2025-09-07T09:17:41.8020277Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8020462Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8020521Z     return mod(*inputs)
2025-09-07T09:17:41.8020791Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8020852Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8021113Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8021208Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8021421Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8021492Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8021747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8021856Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8021859Z 
2025-09-07T09:17:41.8021948Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8022171Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8022228Z     return mod(*inputs)
2025-09-07T09:17:41.8022486Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8022560Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8022817Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8022886Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8023098Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8023164Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8023424Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8023523Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8023767Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.8023832Z     return self.act(input)
2025-09-07T09:17:41.8023836Z 
2025-09-07T09:17:41.8023931Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8024114Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8024172Z     return mod(*inputs)
2025-09-07T09:17:41.8024439Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8024501Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8024760Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8024823Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8025036Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8025110Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8025367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.8025447Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.8025450Z 
2025-09-07T09:17:41.8025539Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8025720Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8025823Z     return mod(*inputs)
2025-09-07T09:17:41.8026083Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8026149Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8026405Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8026476Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8026686Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8026793Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8027054Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8027130Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8027396Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.8027492Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.8027495Z 
2025-09-07T09:17:41.8027584Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8027771Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8027829Z     return mod(*inputs)
2025-09-07T09:17:41.8028136Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8028205Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8028468Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8028530Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8028744Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8028816Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8029072Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8029152Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8029448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.8029598Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8029609Z 
2025-09-07T09:17:41.8029699Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8029880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8029944Z     return mod(*inputs)
2025-09-07T09:17:41.8030209Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8030278Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8030533Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8030596Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8030814Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8030881Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8031140Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8031217Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8031475Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.8031672Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8031675Z 
2025-09-07T09:17:41.8031748Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8031825Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8031893Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8031960Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8032059Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8032247Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8032310Z     return mod(*inputs)
2025-09-07T09:17:41.8032606Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8032674Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8032933Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8032997Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8033213Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8033280Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8033539Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8033652Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8033914Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8034003Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8034294Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.8034418Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.8034421Z 
2025-09-07T09:17:41.8034510Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8034698Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8034756Z     return mod(*inputs)
2025-09-07T09:17:41.8035016Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8035123Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8035381Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8035450Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8035664Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8035729Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8035993Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8036071Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8036334Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8036416Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8036704Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.8036803Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.8036808Z 
2025-09-07T09:17:41.8036897Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8037085Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8037143Z     return mod(*inputs)
2025-09-07T09:17:41.8037459Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8037523Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8037781Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8037850Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8038062Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8038134Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8038393Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8038503Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8038765Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.8038836Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.8038839Z 
2025-09-07T09:17:41.8038934Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8039117Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8039177Z     return mod(*inputs)
2025-09-07T09:17:41.8039440Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8039542Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8039809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8039874Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8040090Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8040157Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8040414Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8040524Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8040528Z 
2025-09-07T09:17:41.8040617Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8040814Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8040909Z     return mod(*inputs)
2025-09-07T09:17:41.8041187Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8041265Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8041541Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8041615Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8041840Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8041924Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8042188Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8042305Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8042520Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.8042585Z     return self.act(input)
2025-09-07T09:17:41.8042588Z 
2025-09-07T09:17:41.8042693Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8042891Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8042954Z     return mod(*inputs)
2025-09-07T09:17:41.8043220Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8043323Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8043587Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8043652Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8043871Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8043950Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8044208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.8044317Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.8044321Z 
2025-09-07T09:17:41.8044416Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8044607Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8044665Z     return mod(*inputs)
2025-09-07T09:17:41.8044927Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8044996Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8045253Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8045321Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8045565Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8045640Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8045895Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8045982Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8046250Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.8046356Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.8046359Z 
2025-09-07T09:17:41.8046459Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8046651Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8046709Z     return mod(*inputs)
2025-09-07T09:17:41.8047067Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8047132Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8047399Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8047463Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8047684Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8047752Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8048014Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8048100Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8048359Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.8048522Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8048526Z 
2025-09-07T09:17:41.8048625Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8048822Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8048890Z     return mod(*inputs)
2025-09-07T09:17:41.8049149Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8049256Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8049513Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8049585Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8049802Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8049871Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8050132Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8050242Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8050507Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.8050663Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8050666Z 
2025-09-07T09:17:41.8050738Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8050809Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8050875Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8050947Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8051035Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8051251Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8051312Z     return mod(*inputs)
2025-09-07T09:17:41.8051575Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8051642Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8051897Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8051965Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8052177Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8052242Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8052499Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8052612Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8052879Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8052973Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8053270Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.8053398Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.8053403Z 
2025-09-07T09:17:41.8053497Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8053686Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8053743Z     return mod(*inputs)
2025-09-07T09:17:41.8054011Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8054080Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8054339Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8054409Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8054620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8054692Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8054950Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8055064Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8055366Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8055456Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8055752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.8055853Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.8055888Z 
2025-09-07T09:17:41.8055983Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8056166Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8056223Z     return mod(*inputs)
2025-09-07T09:17:41.8056492Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8056557Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8056818Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8056880Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8057094Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8057197Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8057461Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8057589Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8057849Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.8057922Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.8057930Z 
2025-09-07T09:17:41.8058018Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8058201Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8058261Z     return mod(*inputs)
2025-09-07T09:17:41.8058523Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8058639Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8058901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8058966Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8059184Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8059251Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8059515Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.8059586Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.8059589Z 
2025-09-07T09:17:41.8059677Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8059866Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8059923Z     return mod(*inputs)
2025-09-07T09:17:41.8060193Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8060259Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8060524Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8060588Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8060800Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8060908Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8061166Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8061280Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8061283Z 
2025-09-07T09:17:41.8061372Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8061554Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8061617Z     return mod(*inputs)
2025-09-07T09:17:41.8061910Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8061982Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8062242Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8062309Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8062527Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8062592Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8062856Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8062990Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8063206Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.8063269Z     return self.act(input)
2025-09-07T09:17:41.8063272Z 
2025-09-07T09:17:41.8063361Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8063554Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8063611Z     return mod(*inputs)
2025-09-07T09:17:41.8063881Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8063943Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8064198Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8064267Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8064511Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8064589Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8064846Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.8064924Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.8064927Z 
2025-09-07T09:17:41.8065020Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8065534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8065635Z     return mod(*inputs)
2025-09-07T09:17:41.8065956Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8066034Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8070207Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8070289Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8070537Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8070618Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8070900Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8071077Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8071346Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.8071457Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.8071461Z 
2025-09-07T09:17:41.8071560Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8071766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8071826Z     return mod(*inputs)
2025-09-07T09:17:41.8072097Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8072209Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8072469Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8072543Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8072766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8072837Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8073099Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8073180Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8073483Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.8073647Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8073651Z 
2025-09-07T09:17:41.8073749Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8073940Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8073999Z     return mod(*inputs)
2025-09-07T09:17:41.8074266Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8074331Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8074592Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8074654Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8075116Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8075191Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8075453Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8075536Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8075790Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.8075955Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8075959Z 
2025-09-07T09:17:41.8076032Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8076100Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8076171Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8076240Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8076334Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8076518Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8076576Z     return mod(*inputs)
2025-09-07T09:17:41.8076842Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8076904Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8077204Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8077269Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8077480Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8077553Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8077809Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8077892Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8078152Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8078282Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8078579Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.8078708Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.8078711Z 
2025-09-07T09:17:41.8078810Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8078994Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8079053Z     return mod(*inputs)
2025-09-07T09:17:41.8079351Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8079419Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8079681Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8079744Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8079963Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8080034Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8080296Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8080373Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8080629Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8080750Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8081041Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.8081151Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.8081154Z 
2025-09-07T09:17:41.8081243Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8081428Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8081491Z     return mod(*inputs)
2025-09-07T09:17:41.8081752Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8081820Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8082077Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8082152Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8082367Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8082436Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8082701Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8082778Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8083080Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.8083151Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.8083155Z 
2025-09-07T09:17:41.8083247Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8083440Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8083497Z     return mod(*inputs)
2025-09-07T09:17:41.8083766Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8083829Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8084128Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8084195Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8084418Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8084491Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8084747Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8084863Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8084866Z 
2025-09-07T09:17:41.8084957Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8085176Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8085237Z     return mod(*inputs)
2025-09-07T09:17:41.8085497Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8085569Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8085825Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8085890Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8086107Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8086173Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8086432Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8086565Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8086782Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.8086846Z     return self.act(input)
2025-09-07T09:17:41.8086849Z 
2025-09-07T09:17:41.8086937Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8087124Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8087184Z     return mod(*inputs)
2025-09-07T09:17:41.8087456Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8087518Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8087772Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8087837Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8088050Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8088125Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8088382Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.8088458Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.8088461Z 
2025-09-07T09:17:41.8088549Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8088766Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8088827Z     return mod(*inputs)
2025-09-07T09:17:41.8089085Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8089153Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8089410Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8089472Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8089687Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8089784Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8090044Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8090121Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8090377Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 315, in forward
2025-09-07T09:17:41.8090485Z     query_states = self.q_proj(hidden_states) * self.scaling
2025-09-07T09:17:41.8090489Z 
2025-09-07T09:17:41.8090577Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8090801Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8090859Z     return mod(*inputs)
2025-09-07T09:17:41.8091124Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8091188Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8091442Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8091510Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8091722Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8091791Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8092059Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8092137Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8092460Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 336, in forward
2025-09-07T09:17:41.8092619Z     key_states = self.k_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8092626Z 
2025-09-07T09:17:41.8092722Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8092907Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8092975Z     return mod(*inputs)
2025-09-07T09:17:41.8093237Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8093297Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8093560Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8093621Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8093841Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8093909Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8094167Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8094248Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8094504Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 337, in forward
2025-09-07T09:17:41.8094697Z     value_states = self.v_proj(current_states).view(bsz, -1, self.num_heads, self.head_dim)
2025-09-07T09:17:41.8094701Z 
2025-09-07T09:17:41.8094776Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8094848Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8094917Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8094984Z cudagraph partition due to non gpu ops
2025-09-07T09:17:41.8095079Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8095263Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8095360Z     return mod(*inputs)
2025-09-07T09:17:41.8095620Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8095685Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8095947Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8096010Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8096225Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8096291Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8096583Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8096665Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8096922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8097013Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8097305Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward
2025-09-07T09:17:41.8097433Z     attn_output = torch.nn.functional.scaled_dot_product_attention(
2025-09-07T09:17:41.8097436Z 
2025-09-07T09:17:41.8097604Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8097796Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8097859Z     return mod(*inputs)
2025-09-07T09:17:41.8098163Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8098234Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8098494Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8098561Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8098776Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8098848Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8099109Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8099188Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8099452Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 351, in forward
2025-09-07T09:17:41.8099538Z     attn_output, attn_weights = attention_interface(
2025-09-07T09:17:41.8099828Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward
2025-09-07T09:17:41.8099932Z     attn_output = attn_output.transpose(1, 2).contiguous()
2025-09-07T09:17:41.8099936Z 
2025-09-07T09:17:41.8100026Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8100215Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8100308Z     return mod(*inputs)
2025-09-07T09:17:41.8100571Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8100641Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8100901Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8100968Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8101183Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8101254Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8101551Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 410, in forward
2025-09-07T09:17:41.8101628Z     hidden_states, attn_weights = self.self_attn(
2025-09-07T09:17:41.8101888Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 365, in forward
2025-09-07T09:17:41.8101961Z     attn_output = self.out_proj(attn_output)
2025-09-07T09:17:41.8101965Z 
2025-09-07T09:17:41.8102060Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8102240Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8102297Z     return mod(*inputs)
2025-09-07T09:17:41.8102593Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8102657Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8102922Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8102985Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8103385Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8103463Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8103732Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 417, in forward
2025-09-07T09:17:41.8103805Z     hidden_states = residual + hidden_states
2025-09-07T09:17:41.8103808Z 
2025-09-07T09:17:41.8103898Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8104176Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8104233Z     return mod(*inputs)
2025-09-07T09:17:41.8104500Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8104569Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8104830Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8104899Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8105114Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8105182Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8105448Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8105555Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8105560Z 
2025-09-07T09:17:41.8105657Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8105845Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8105907Z     return mod(*inputs)
2025-09-07T09:17:41.8106170Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8106232Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8106556Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8106621Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8106843Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8106911Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8107171Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 421, in forward
2025-09-07T09:17:41.8107278Z     hidden_states = self.activation_fn(self.fc1(hidden_states))
2025-09-07T09:17:41.8107536Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward
2025-09-07T09:17:41.8107604Z     return self.act(input)
2025-09-07T09:17:41.8107607Z 
2025-09-07T09:17:41.8107698Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8107891Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8107947Z     return mod(*inputs)
2025-09-07T09:17:41.8108208Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1596, in forward
2025-09-07T09:17:41.8108274Z     encoder_outputs = self.encoder(
2025-09-07T09:17:41.8108534Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 708, in forward
2025-09-07T09:17:41.8108646Z     layer_outputs = encoder_layer(
2025-09-07T09:17:41.8108868Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__
2025-09-07T09:17:41.8108940Z     return super().__call__(*args, **kwargs)
2025-09-07T09:17:41.8109210Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 423, in forward
2025-09-07T09:17:41.8109284Z     hidden_states = self.fc2(hidden_states)
2025-09-07T09:17:41.8109288Z 
2025-09-07T09:17:41.8109389Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8109580Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8109636Z     return mod(*inputs)
2025-09-07T09:17:41.8109904Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1612, in forward
2025-09-07T09:17:41.8110021Z     hidden_states = self.projector(hidden_states)
2025-09-07T09:17:41.8110025Z 
2025-09-07T09:17:41.8110121Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:17:41.8110304Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:17:41.8110368Z     return mod(*inputs)
2025-09-07T09:17:41.8110628Z   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/whisper/modeling_whisper.py", line 1615, in forward
2025-09-07T09:17:41.8110704Z     logits = self.classifier(pooled_output)
2025-09-07T09:17:41.8110707Z 
2025-09-07T09:18:03.2735482Z 
2025-09-07T09:18:03.8841022Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:18:04.4888439Z running benchmark:   3% 1/30 [00:00<00:17,  1.64it/s]
2025-09-07T09:18:05.0968109Z running benchmark:   7% 2/30 [00:01<00:16,  1.65it/s]
2025-09-07T09:18:05.7181268Z running benchmark:  10% 3/30 [00:01<00:16,  1.65it/s]
2025-09-07T09:18:06.3432068Z running benchmark:  13% 4/30 [00:02<00:15,  1.63it/s]
2025-09-07T09:18:06.9615282Z running benchmark:  17% 5/30 [00:03<00:15,  1.62it/s]
2025-09-07T09:18:07.5861948Z running benchmark:  20% 6/30 [00:03<00:14,  1.62it/s]
2025-09-07T09:18:08.2130240Z running benchmark:  23% 7/30 [00:04<00:14,  1.61it/s]
2025-09-07T09:18:08.8316011Z running benchmark:  27% 8/30 [00:04<00:13,  1.61it/s]
2025-09-07T09:18:09.4498080Z running benchmark:  30% 9/30 [00:05<00:13,  1.61it/s]
2025-09-07T09:18:10.0678626Z running benchmark:  33% 10/30 [00:06<00:12,  1.61it/s]
2025-09-07T09:18:10.6896455Z running benchmark:  37% 11/30 [00:06<00:11,  1.61it/s]
2025-09-07T09:18:11.3514084Z running benchmark:  40% 12/30 [00:07<00:11,  1.61it/s]
2025-09-07T09:18:11.9647497Z running benchmark:  43% 13/30 [00:08<00:10,  1.58it/s]
2025-09-07T09:18:12.5714694Z running benchmark:  47% 14/30 [00:08<00:10,  1.60it/s]
2025-09-07T09:18:13.1900872Z running benchmark:  50% 15/30 [00:09<00:09,  1.61it/s]
2025-09-07T09:18:13.8080165Z running benchmark:  53% 16/30 [00:09<00:08,  1.61it/s]
2025-09-07T09:18:14.4307550Z running benchmark:  57% 17/30 [00:10<00:08,  1.61it/s]
2025-09-07T09:18:15.0532239Z running benchmark:  60% 18/30 [00:11<00:07,  1.61it/s]
2025-09-07T09:18:15.6772293Z running benchmark:  63% 19/30 [00:11<00:06,  1.61it/s]
2025-09-07T09:18:16.2990298Z running benchmark:  67% 20/30 [00:12<00:06,  1.61it/s]
2025-09-07T09:18:16.9250198Z running benchmark:  70% 21/30 [00:13<00:05,  1.61it/s]
2025-09-07T09:18:17.5480575Z running benchmark:  73% 22/30 [00:13<00:04,  1.60it/s]
2025-09-07T09:18:18.1649732Z running benchmark:  77% 23/30 [00:14<00:04,  1.60it/s]
2025-09-07T09:18:18.7900328Z running benchmark:  80% 24/30 [00:14<00:03,  1.61it/s]
2025-09-07T09:18:19.4108746Z running benchmark:  83% 25/30 [00:15<00:03,  1.61it/s]
2025-09-07T09:18:20.0333168Z running benchmark:  87% 26/30 [00:16<00:02,  1.61it/s]
2025-09-07T09:18:20.6569572Z running benchmark:  90% 27/30 [00:16<00:01,  1.61it/s]
2025-09-07T09:18:21.2833640Z running benchmark:  93% 28/30 [00:17<00:01,  1.61it/s]
2025-09-07T09:18:21.9059755Z running benchmark:  97% 29/30 [00:18<00:00,  1.60it/s]
2025-09-07T09:18:21.9060920Z running benchmark: 100% 30/30 [00:18<00:00,  1.60it/s]
2025-09-07T09:18:21.9061895Z running benchmark: 100% 30/30 [00:18<00:00,  1.61it/s]
2025-09-07T09:18:22.0590363Z 1.194x
2025-09-07T09:18:22.0594572Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:18:24.6938389Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:18:24.6939410Z   import pynvml  # type: ignore[import]
2025-09-07T09:18:26.7283363Z 
2025-09-07T09:18:26.9221782Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:18:26.9222066Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:18:26.9223810Z cpu  eval  lennard_jones                      
2025-09-07T09:18:29.0389220Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0389669Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0390018Z     return mod(*inputs)
2025-09-07T09:18:29.0390128Z 
2025-09-07T09:18:29.0390249Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0390620Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0390958Z     return mod(*inputs)
2025-09-07T09:18:29.0391072Z 
2025-09-07T09:18:29.0391176Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0391537Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0391858Z     return mod(*inputs)
2025-09-07T09:18:29.0391953Z 
2025-09-07T09:18:29.0392054Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0392417Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0392742Z     return mod(*inputs)
2025-09-07T09:18:29.0392835Z 
2025-09-07T09:18:29.0392941Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0393285Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0393592Z     return mod(*inputs)
2025-09-07T09:18:29.0393693Z 
2025-09-07T09:18:29.0393783Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0394216Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0394530Z     return mod(*inputs)
2025-09-07T09:18:29.0394620Z 
2025-09-07T09:18:29.0394716Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0395050Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0395357Z     return mod(*inputs)
2025-09-07T09:18:29.0395453Z 
2025-09-07T09:18:29.0395546Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0395880Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0396261Z     return mod(*inputs)
2025-09-07T09:18:29.0396359Z 
2025-09-07T09:18:29.0396449Z cudagraph partition due to non gpu ops. Found from : 
2025-09-07T09:18:29.0396793Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:18:29.0397112Z     return mod(*inputs)
2025-09-07T09:18:29.0397199Z 
2025-09-07T09:18:33.7523725Z 
2025-09-07T09:18:33.8102750Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:18:33.8103189Z running benchmark: 100% 30/30 [00:00<00:00, 519.31it/s]
2025-09-07T09:18:33.8560770Z 0.819x
2025-09-07T09:18:33.8563110Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:18:35.3360026Z speedup             gmean=1.21x mean=1.250x
2025-09-07T09:18:35.3360736Z abs_latency         gmean=46.21x mean=135.163x
2025-09-07T09:18:35.3361081Z compilation_latency mean=22.365 seconds
2025-09-07T09:18:35.3361667Z compression_ratio   mean=0.903x
2025-09-07T09:18:35.3364587Z eager_peak_mem      gmean=205.62x mean=331.602x
2025-09-07T09:18:35.3366874Z dynamo_peak_mem     gmean=228.33x mean=350.687x
2025-09-07T09:18:35.3369045Z calls_captured      gmean=273.07x mean=611.609x
2025-09-07T09:18:35.3371171Z unique_graphs       gmean=1.66x mean=3.043x
2025-09-07T09:18:35.3373287Z graph_breaks        gmean=0.00x mean=2.174x
2025-09-07T09:18:35.3375409Z unique_graph_breaks gmean=0.00x mean=0.565x
2025-09-07T09:18:35.3377584Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T09:18:35.3379750Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T09:18:35.3381778Z cudagraph_skips     gmean=1.56x mean=2.522x
2025-09-07T09:18:35.9169543Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *cppwrapper-true* ]]
2025-09-07T09:18:35.9170326Z + TORCHINDUCTOR_CPP_WRAPPER=1
2025-09-07T09:18:35.9171284Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv
2025-09-07T09:18:36.3130936Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:18:36.3132034Z   import pynvml  # type: ignore[import]
2025-09-07T09:18:38.7679122Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:18:38.7680014Z   import pynvml  # type: ignore[import]
2025-09-07T09:18:40.7877203Z 
2025-09-07T09:18:46.4334303Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:18:46.4334708Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:18:46.4537680Z cpu  eval  dlrm                               
2025-09-07T09:18:58.0396712Z 
2025-09-07T09:18:58.1787698Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:18:58.3143264Z running benchmark:   7% 2/30 [00:00<00:01, 14.42it/s]
2025-09-07T09:18:58.4505502Z running benchmark:  13% 4/30 [00:00<00:01, 14.61it/s]
2025-09-07T09:18:58.5863145Z running benchmark:  20% 6/30 [00:00<00:01, 14.64it/s]
2025-09-07T09:18:58.7229866Z running benchmark:  27% 8/30 [00:00<00:01, 14.68it/s]
2025-09-07T09:18:58.8594273Z running benchmark:  33% 10/30 [00:00<00:01, 14.66it/s]
2025-09-07T09:18:58.9950326Z running benchmark:  40% 12/30 [00:00<00:01, 14.66it/s]
2025-09-07T09:18:59.1317406Z running benchmark:  47% 14/30 [00:00<00:01, 14.69it/s]
2025-09-07T09:18:59.2671623Z running benchmark:  53% 16/30 [00:01<00:00, 14.67it/s]
2025-09-07T09:18:59.4034421Z running benchmark:  60% 18/30 [00:01<00:00, 14.70it/s]
2025-09-07T09:18:59.5395659Z running benchmark:  67% 20/30 [00:01<00:00, 14.70it/s]
2025-09-07T09:18:59.6751804Z running benchmark:  73% 22/30 [00:01<00:00, 14.69it/s]
2025-09-07T09:18:59.8112751Z running benchmark:  80% 24/30 [00:01<00:00, 14.71it/s]
2025-09-07T09:18:59.9475601Z running benchmark:  87% 26/30 [00:01<00:00, 14.71it/s]
2025-09-07T09:19:00.0831247Z running benchmark:  93% 28/30 [00:01<00:00, 14.70it/s]
2025-09-07T09:19:00.0831595Z running benchmark: 100% 30/30 [00:02<00:00, 14.72it/s]
2025-09-07T09:19:00.0831878Z running benchmark: 100% 30/30 [00:02<00:00, 14.68it/s]
2025-09-07T09:19:00.1835511Z 0.970x
2025-09-07T09:19:00.1838506Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:19:02.2908738Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:19:02.2909641Z   import pynvml  # type: ignore[import]
2025-09-07T09:19:04.3115517Z 
2025-09-07T09:19:06.9468581Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T09:19:07.1885906Z 
2025-09-07T09:19:07.1886383Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:19:07.1886772Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T09:19:07.1887226Z cpu  eval  doctr_det_predictor                
2025-09-07T09:19:07.1887551Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2025-09-07T09:19:36.0051176Z 
2025-09-07T09:19:36.2085251Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:19:36.4004310Z running benchmark:   3% 1/30 [00:00<00:05,  4.92it/s]
2025-09-07T09:19:36.5913254Z running benchmark:   7% 2/30 [00:00<00:05,  5.09it/s]
2025-09-07T09:19:36.7822902Z running benchmark:  10% 3/30 [00:00<00:05,  5.16it/s]
2025-09-07T09:19:36.9729184Z running benchmark:  13% 4/30 [00:00<00:05,  5.19it/s]
2025-09-07T09:19:37.1621910Z running benchmark:  17% 5/30 [00:00<00:04,  5.21it/s]
2025-09-07T09:19:37.3520580Z running benchmark:  20% 6/30 [00:01<00:04,  5.23it/s]
2025-09-07T09:19:37.5415965Z running benchmark:  23% 7/30 [00:01<00:04,  5.24it/s]
2025-09-07T09:19:37.7339577Z running benchmark:  27% 8/30 [00:01<00:04,  5.25it/s]
2025-09-07T09:19:37.9251116Z running benchmark:  30% 9/30 [00:01<00:04,  5.24it/s]
2025-09-07T09:19:38.1152228Z running benchmark:  33% 10/30 [00:01<00:03,  5.24it/s]
2025-09-07T09:19:38.3051824Z running benchmark:  37% 11/30 [00:02<00:03,  5.24it/s]
2025-09-07T09:19:38.5031539Z running benchmark:  40% 12/30 [00:02<00:03,  5.25it/s]
2025-09-07T09:19:38.6943561Z running benchmark:  43% 13/30 [00:02<00:03,  5.19it/s]
2025-09-07T09:19:38.8848054Z running benchmark:  47% 14/30 [00:02<00:03,  5.20it/s]
2025-09-07T09:19:39.0766888Z running benchmark:  50% 15/30 [00:02<00:02,  5.22it/s]
2025-09-07T09:19:39.2693937Z running benchmark:  53% 16/30 [00:03<00:02,  5.21it/s]
2025-09-07T09:19:39.4614578Z running benchmark:  57% 17/30 [00:03<00:02,  5.21it/s]
2025-09-07T09:19:39.6521674Z running benchmark:  60% 18/30 [00:03<00:02,  5.21it/s]
2025-09-07T09:19:39.8439039Z running benchmark:  63% 19/30 [00:03<00:02,  5.22it/s]
2025-09-07T09:19:40.0359215Z running benchmark:  67% 20/30 [00:03<00:01,  5.22it/s]
2025-09-07T09:19:40.2370998Z running benchmark:  70% 21/30 [00:04<00:01,  5.21it/s]
2025-09-07T09:19:40.4362498Z running benchmark:  73% 22/30 [00:04<00:01,  5.14it/s]
2025-09-07T09:19:40.6260204Z running benchmark:  77% 23/30 [00:04<00:01,  5.10it/s]
2025-09-07T09:19:40.8159885Z running benchmark:  80% 24/30 [00:04<00:01,  5.15it/s]
2025-09-07T09:19:41.0071790Z running benchmark:  83% 25/30 [00:04<00:00,  5.18it/s]
2025-09-07T09:19:41.1991916Z running benchmark:  87% 26/30 [00:05<00:00,  5.20it/s]
2025-09-07T09:19:41.3920430Z running benchmark:  90% 27/30 [00:05<00:00,  5.20it/s]
2025-09-07T09:19:41.5900098Z running benchmark:  93% 28/30 [00:05<00:00,  5.20it/s]
2025-09-07T09:19:41.7832454Z running benchmark:  97% 29/30 [00:05<00:00,  5.15it/s]
2025-09-07T09:19:41.7833092Z running benchmark: 100% 30/30 [00:05<00:00,  5.16it/s]
2025-09-07T09:19:41.7833725Z running benchmark: 100% 30/30 [00:05<00:00,  5.19it/s]
2025-09-07T09:19:41.9055956Z 2.139x
2025-09-07T09:19:41.9060068Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:19:44.6915710Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:19:44.6917026Z   import pynvml  # type: ignore[import]
2025-09-07T09:19:46.7073504Z 
2025-09-07T09:19:49.2972622Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T09:19:49.3307612Z 
2025-09-07T09:19:49.3307807Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:19:49.3308195Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T09:19:49.3310431Z cpu  eval  doctr_reco_predictor               
2025-09-07T09:19:49.3310823Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2025-09-07T09:20:02.4478486Z 
2025-09-07T09:20:02.5658576Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:20:02.6781004Z running benchmark:  20% 6/30 [00:00<00:00, 51.04it/s]
2025-09-07T09:20:02.7907228Z running benchmark:  40% 12/30 [00:00<00:00, 52.43it/s]
2025-09-07T09:20:02.9034299Z running benchmark:  60% 18/30 [00:00<00:00, 52.81it/s]
2025-09-07T09:20:03.0171395Z running benchmark:  80% 24/30 [00:00<00:00, 52.97it/s]
2025-09-07T09:20:03.0171733Z running benchmark: 100% 30/30 [00:00<00:00, 53.01it/s]
2025-09-07T09:20:03.0172017Z running benchmark: 100% 30/30 [00:00<00:00, 52.78it/s]
2025-09-07T09:20:03.0885338Z 1.483x
2025-09-07T09:20:03.0888908Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:20:05.3609260Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:20:05.3610138Z   import pynvml  # type: ignore[import]
2025-09-07T09:20:07.3803182Z 
2025-09-07T09:20:07.4328714Z loading model: 0it [00:00, ?it/s]Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
2025-09-07T09:20:07.4329415Z Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
2025-09-07T09:20:07.4330021Z Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
2025-09-07T09:20:07.4330603Z See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
2025-09-07T09:20:08.2813624Z 
2025-09-07T09:20:08.2814726Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:20:08.2828189Z cpu  eval  drq                                
2025-09-07T09:20:17.5728284Z 
2025-09-07T09:20:17.6751729Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:20:17.7621619Z running benchmark:  53% 16/30 [00:00<00:00, 156.75it/s]
2025-09-07T09:20:17.7621960Z running benchmark: 100% 30/30 [00:00<00:00, 158.99it/s]
2025-09-07T09:20:17.8194447Z 0.782x
2025-09-07T09:20:17.8197738Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:20:19.9105615Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:20:19.9106903Z   import pynvml  # type: ignore[import]
2025-09-07T09:20:21.9312295Z 
2025-09-07T09:20:23.0220568Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:20:23.0221163Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:20:23.0386233Z cpu  eval  fastNLP_Bert                       
2025-09-07T09:20:23.7672136Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T09:20:23.7672832Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T09:20:23.7673768Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T09:20:23.7674286Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T09:20:23.7674875Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T09:20:23.7675361Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:20:23.7675821Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T09:20:23.7676505Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 482, in forward_pass
2025-09-07T09:20:23.7677303Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(*inputs)
2025-09-07T09:20:23.7677985Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/models/bert.py", line 265, in forward
2025-09-07T09:20:23.7678667Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     sequence_output = self.bert(words)
2025-09-07T09:20:23.7679381Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2025-09-07T09:20:23.7680078Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.model(words)
2025-09-07T09:20:23.7680761Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2025-09-07T09:20:23.7682044Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2025-09-07T09:20:23.7682657Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:20:23.7683035Z W0907 09:20:23.766470 189599 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:20:43.0384353Z 
2025-09-07T09:20:43.1516177Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:20:43.2628531Z running benchmark:   3% 1/30 [00:00<00:03,  8.87it/s]
2025-09-07T09:20:43.3744615Z running benchmark:   7% 2/30 [00:00<00:03,  8.93it/s]
2025-09-07T09:20:43.4862233Z running benchmark:  10% 3/30 [00:00<00:03,  8.95it/s]
2025-09-07T09:20:43.5975447Z running benchmark:  13% 4/30 [00:00<00:02,  8.95it/s]
2025-09-07T09:20:43.7084822Z running benchmark:  17% 5/30 [00:00<00:02,  8.96it/s]
2025-09-07T09:20:43.8195937Z running benchmark:  20% 6/30 [00:00<00:02,  8.98it/s]
2025-09-07T09:20:43.9315381Z running benchmark:  23% 7/30 [00:00<00:02,  8.99it/s]
2025-09-07T09:20:44.0424379Z running benchmark:  27% 8/30 [00:00<00:02,  8.97it/s]
2025-09-07T09:20:44.1541930Z running benchmark:  30% 9/30 [00:01<00:02,  8.98it/s]
2025-09-07T09:20:44.2661375Z running benchmark:  33% 10/30 [00:01<00:02,  8.97it/s]
2025-09-07T09:20:44.3773536Z running benchmark:  37% 11/30 [00:01<00:02,  8.96it/s]
2025-09-07T09:20:44.4890811Z running benchmark:  40% 12/30 [00:01<00:02,  8.97it/s]
2025-09-07T09:20:44.6003430Z running benchmark:  43% 13/30 [00:01<00:01,  8.96it/s]
2025-09-07T09:20:44.7125181Z running benchmark:  47% 14/30 [00:01<00:01,  8.97it/s]
2025-09-07T09:20:44.8236673Z running benchmark:  50% 15/30 [00:01<00:01,  8.95it/s]
2025-09-07T09:20:44.9357956Z running benchmark:  53% 16/30 [00:01<00:01,  8.97it/s]
2025-09-07T09:20:45.0475398Z running benchmark:  57% 17/30 [00:01<00:01,  8.95it/s]
2025-09-07T09:20:45.1774989Z running benchmark:  60% 18/30 [00:02<00:01,  8.95it/s]
2025-09-07T09:20:45.2964232Z running benchmark:  63% 19/30 [00:02<00:01,  8.53it/s]
2025-09-07T09:20:45.4092917Z running benchmark:  67% 20/30 [00:02<00:01,  8.49it/s]
2025-09-07T09:20:45.5215383Z running benchmark:  70% 21/30 [00:02<00:01,  8.60it/s]
2025-09-07T09:20:45.6325871Z running benchmark:  73% 22/30 [00:02<00:00,  8.69it/s]
2025-09-07T09:20:45.7434093Z running benchmark:  77% 23/30 [00:02<00:00,  8.78it/s]
2025-09-07T09:20:45.8543113Z running benchmark:  80% 24/30 [00:02<00:00,  8.85it/s]
2025-09-07T09:20:45.9649940Z running benchmark:  83% 25/30 [00:02<00:00,  8.90it/s]
2025-09-07T09:20:46.0776868Z running benchmark:  87% 26/30 [00:02<00:00,  8.94it/s]
2025-09-07T09:20:46.1881107Z running benchmark:  90% 27/30 [00:03<00:00,  8.92it/s]
2025-09-07T09:20:46.2998875Z running benchmark:  93% 28/30 [00:03<00:00,  8.96it/s]
2025-09-07T09:20:46.4113250Z running benchmark:  97% 29/30 [00:03<00:00,  8.96it/s]
2025-09-07T09:20:46.4114192Z running benchmark: 100% 30/30 [00:03<00:00,  8.96it/s]
2025-09-07T09:20:46.4114984Z running benchmark: 100% 30/30 [00:03<00:00,  8.89it/s]
2025-09-07T09:20:46.4932434Z 1.662x
2025-09-07T09:20:46.4936172Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:20:48.8443498Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:20:48.8444395Z   import pynvml  # type: ignore[import]
2025-09-07T09:20:50.8631188Z 
2025-09-07T09:20:51.2701093Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:20:51.2701375Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:20:51.2728164Z cpu  eval  functorch_dp_cifar10               
2025-09-07T09:21:04.7971869Z 
2025-09-07T09:21:04.8986848Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:21:05.0165608Z running benchmark:  17% 5/30 [00:00<00:00, 49.47it/s]
2025-09-07T09:21:05.1335939Z running benchmark:  37% 11/30 [00:00<00:00, 50.33it/s]
2025-09-07T09:21:05.2501247Z running benchmark:  57% 17/30 [00:00<00:00, 50.78it/s]
2025-09-07T09:21:05.3667653Z running benchmark:  77% 23/30 [00:00<00:00, 51.06it/s]
2025-09-07T09:21:05.4599246Z running benchmark:  97% 29/30 [00:00<00:00, 51.20it/s]
2025-09-07T09:21:05.4599591Z running benchmark: 100% 30/30 [00:00<00:00, 45.29it/s]
2025-09-07T09:21:05.5163661Z 0.815x
2025-09-07T09:21:05.5166572Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:21:07.6571913Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:21:07.6572793Z   import pynvml  # type: ignore[import]
2025-09-07T09:21:09.6735956Z 
2025-09-07T09:21:09.8559109Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:21:09.8559404Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:21:09.8562706Z cpu  eval  functorch_maml_omniglot            
2025-09-07T09:21:18.5505412Z 
2025-09-07T09:21:18.6510120Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:21:18.6510453Z running benchmark: 100% 30/30 [00:00<00:00, 299.77it/s]
2025-09-07T09:21:18.6965924Z 0.999x
2025-09-07T09:21:18.6969333Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:21:20.6644114Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:21:20.6644997Z   import pynvml  # type: ignore[import]
2025-09-07T09:21:22.6829228Z 
2025-09-07T09:21:24.2271358Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:21:24.2271672Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:21:24.2300213Z cpu  eval  hf_Albert                          
2025-09-07T09:21:39.0512805Z 
2025-09-07T09:21:39.2382264Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:21:39.4206383Z running benchmark:   7% 2/30 [00:00<00:02, 10.72it/s]
2025-09-07T09:21:39.6038664Z running benchmark:  13% 4/30 [00:00<00:02, 10.86it/s]
2025-09-07T09:21:39.7829263Z running benchmark:  20% 6/30 [00:00<00:02, 10.89it/s]
2025-09-07T09:21:39.9644604Z running benchmark:  27% 8/30 [00:00<00:02, 11.00it/s]
2025-09-07T09:21:40.1465590Z running benchmark:  33% 10/30 [00:00<00:01, 11.00it/s]
2025-09-07T09:21:40.3260611Z running benchmark:  40% 12/30 [00:01<00:01, 11.00it/s]
2025-09-07T09:21:40.5069995Z running benchmark:  47% 14/30 [00:01<00:01, 11.04it/s]
2025-09-07T09:21:40.6907258Z running benchmark:  53% 16/30 [00:01<00:01, 11.05it/s]
2025-09-07T09:21:40.8718920Z running benchmark:  60% 18/30 [00:01<00:01, 11.00it/s]
2025-09-07T09:21:41.0517787Z running benchmark:  67% 20/30 [00:01<00:00, 11.01it/s]
2025-09-07T09:21:41.2323285Z running benchmark:  73% 22/30 [00:02<00:00, 11.04it/s]
2025-09-07T09:21:41.4135246Z running benchmark:  80% 24/30 [00:02<00:00, 11.05it/s]
2025-09-07T09:21:41.5959043Z running benchmark:  87% 26/30 [00:02<00:00, 11.05it/s]
2025-09-07T09:21:41.7758426Z running benchmark:  93% 28/30 [00:02<00:00, 11.02it/s]
2025-09-07T09:21:41.7758808Z running benchmark: 100% 30/30 [00:02<00:00, 11.05it/s]
2025-09-07T09:21:41.7759150Z running benchmark: 100% 30/30 [00:02<00:00, 11.01it/s]
2025-09-07T09:21:41.8399322Z 1.380x
2025-09-07T09:21:41.8403119Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:21:44.2842322Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:21:44.2843191Z   import pynvml  # type: ignore[import]
2025-09-07T09:21:46.3040045Z 
2025-09-07T09:21:50.1615880Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:21:50.1616483Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:21:50.2540752Z cpu  eval  hf_Bart                            
2025-09-07T09:22:11.8117689Z 
2025-09-07T09:22:11.9501267Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:22:12.0859779Z running benchmark:   3% 1/30 [00:00<00:04,  7.25it/s]
2025-09-07T09:22:12.2200644Z running benchmark:   7% 2/30 [00:00<00:03,  7.31it/s]
2025-09-07T09:22:12.3551945Z running benchmark:  10% 3/30 [00:00<00:03,  7.38it/s]
2025-09-07T09:22:12.4900224Z running benchmark:  13% 4/30 [00:00<00:03,  7.39it/s]
2025-09-07T09:22:12.6239567Z running benchmark:  17% 5/30 [00:00<00:03,  7.40it/s]
2025-09-07T09:22:12.7586007Z running benchmark:  20% 6/30 [00:00<00:03,  7.42it/s]
2025-09-07T09:22:12.8920468Z running benchmark:  23% 7/30 [00:00<00:03,  7.42it/s]
2025-09-07T09:22:13.0264622Z running benchmark:  27% 8/30 [00:01<00:02,  7.44it/s]
2025-09-07T09:22:13.1599728Z running benchmark:  30% 9/30 [00:01<00:02,  7.44it/s]
2025-09-07T09:22:13.2944103Z running benchmark:  33% 10/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:22:13.4280906Z running benchmark:  37% 11/30 [00:01<00:02,  7.45it/s]
2025-09-07T09:22:13.5626928Z running benchmark:  40% 12/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:22:13.6961859Z running benchmark:  43% 13/30 [00:01<00:02,  7.45it/s]
2025-09-07T09:22:13.8310368Z running benchmark:  47% 14/30 [00:01<00:02,  7.46it/s]
2025-09-07T09:22:13.9649091Z running benchmark:  50% 15/30 [00:02<00:02,  7.45it/s]
2025-09-07T09:22:14.0990348Z running benchmark:  53% 16/30 [00:02<00:01,  7.46it/s]
2025-09-07T09:22:14.2340452Z running benchmark:  57% 17/30 [00:02<00:01,  7.46it/s]
2025-09-07T09:22:14.3687837Z running benchmark:  60% 18/30 [00:02<00:01,  7.44it/s]
2025-09-07T09:22:14.5029465Z running benchmark:  63% 19/30 [00:02<00:01,  7.43it/s]
2025-09-07T09:22:14.7089813Z running benchmark:  67% 20/30 [00:02<00:01,  7.44it/s]
2025-09-07T09:22:14.8430686Z running benchmark:  70% 21/30 [00:02<00:01,  6.42it/s]
2025-09-07T09:22:14.9772847Z running benchmark:  73% 22/30 [00:03<00:01,  6.70it/s]
2025-09-07T09:22:15.1114802Z running benchmark:  77% 23/30 [00:03<00:01,  6.91it/s]
2025-09-07T09:22:15.2454992Z running benchmark:  80% 24/30 [00:03<00:00,  7.06it/s]
2025-09-07T09:22:15.3809683Z running benchmark:  83% 25/30 [00:03<00:00,  7.18it/s]
2025-09-07T09:22:15.5143134Z running benchmark:  87% 26/30 [00:03<00:00,  7.24it/s]
2025-09-07T09:22:15.6492596Z running benchmark:  90% 27/30 [00:03<00:00,  7.31it/s]
2025-09-07T09:22:15.7829685Z running benchmark:  93% 28/30 [00:03<00:00,  7.34it/s]
2025-09-07T09:22:15.9170137Z running benchmark:  97% 29/30 [00:03<00:00,  7.38it/s]
2025-09-07T09:22:15.9170625Z running benchmark: 100% 30/30 [00:04<00:00,  7.41it/s]
2025-09-07T09:22:15.9170903Z running benchmark: 100% 30/30 [00:04<00:00,  7.31it/s]
2025-09-07T09:22:16.0171227Z 1.225x
2025-09-07T09:22:16.0175309Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:22:18.7271029Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:22:18.7271927Z   import pynvml  # type: ignore[import]
2025-09-07T09:22:20.7323007Z 
2025-09-07T09:22:23.6001087Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:22:23.6001447Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:22:23.6186126Z cpu  eval  hf_Bert                            
2025-09-07T09:22:40.6796656Z 
2025-09-07T09:22:40.7865342Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:22:40.8893377Z running benchmark:   3% 1/30 [00:00<00:03,  9.38it/s]
2025-09-07T09:22:40.9921009Z running benchmark:   7% 2/30 [00:00<00:02,  9.58it/s]
2025-09-07T09:22:41.0941207Z running benchmark:  10% 3/30 [00:00<00:02,  9.65it/s]
2025-09-07T09:22:41.1981759Z running benchmark:  13% 4/30 [00:00<00:02,  9.71it/s]
2025-09-07T09:22:41.3001363Z running benchmark:  17% 5/30 [00:00<00:02,  9.67it/s]
2025-09-07T09:22:41.4030442Z running benchmark:  20% 6/30 [00:00<00:02,  9.72it/s]
2025-09-07T09:22:41.5068968Z running benchmark:  23% 7/30 [00:00<00:02,  9.72it/s]
2025-09-07T09:22:41.6091301Z running benchmark:  27% 8/30 [00:00<00:02,  9.69it/s]
2025-09-07T09:22:41.7116897Z running benchmark:  30% 9/30 [00:00<00:02,  9.72it/s]
2025-09-07T09:22:41.8147184Z running benchmark:  33% 10/30 [00:01<00:02,  9.73it/s]
2025-09-07T09:22:41.9161634Z running benchmark:  37% 11/30 [00:01<00:01,  9.72it/s]
2025-09-07T09:22:42.0192112Z running benchmark:  40% 12/30 [00:01<00:01,  9.76it/s]
2025-09-07T09:22:42.1223143Z running benchmark:  43% 13/30 [00:01<00:01,  9.74it/s]
2025-09-07T09:22:42.2245949Z running benchmark:  47% 14/30 [00:01<00:01,  9.73it/s]
2025-09-07T09:22:42.3271763Z running benchmark:  50% 15/30 [00:01<00:01,  9.74it/s]
2025-09-07T09:22:42.4304703Z running benchmark:  53% 16/30 [00:01<00:01,  9.74it/s]
2025-09-07T09:22:42.5330916Z running benchmark:  57% 17/30 [00:01<00:01,  9.73it/s]
2025-09-07T09:22:42.6362164Z running benchmark:  60% 18/30 [00:01<00:01,  9.73it/s]
2025-09-07T09:22:42.7381175Z running benchmark:  63% 19/30 [00:01<00:01,  9.72it/s]
2025-09-07T09:22:42.8409030Z running benchmark:  67% 20/30 [00:02<00:01,  9.75it/s]
2025-09-07T09:22:42.9434238Z running benchmark:  70% 21/30 [00:02<00:00,  9.74it/s]
2025-09-07T09:22:43.0451546Z running benchmark:  73% 22/30 [00:02<00:00,  9.75it/s]
2025-09-07T09:22:43.1715883Z running benchmark:  77% 23/30 [00:02<00:00,  9.77it/s]
2025-09-07T09:22:43.3192361Z running benchmark:  80% 24/30 [00:02<00:00,  9.13it/s]
2025-09-07T09:22:43.4232806Z running benchmark:  83% 25/30 [00:02<00:00,  8.27it/s]
2025-09-07T09:22:43.5255077Z running benchmark:  87% 26/30 [00:02<00:00,  8.63it/s]
2025-09-07T09:22:43.6286082Z running benchmark:  90% 27/30 [00:02<00:00,  8.95it/s]
2025-09-07T09:22:43.7328162Z running benchmark:  93% 28/30 [00:02<00:00,  9.16it/s]
2025-09-07T09:22:43.8364589Z running benchmark:  97% 29/30 [00:03<00:00,  9.28it/s]
2025-09-07T09:22:43.8364943Z running benchmark: 100% 30/30 [00:03<00:00,  9.39it/s]
2025-09-07T09:22:43.8365227Z running benchmark: 100% 30/30 [00:03<00:00,  9.50it/s]
2025-09-07T09:22:43.9250876Z 1.190x
2025-09-07T09:22:43.9255648Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:22:46.4569709Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:22:46.4570604Z   import pynvml  # type: ignore[import]
2025-09-07T09:22:48.4729551Z 
2025-09-07T09:22:53.0852186Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:22:53.0852548Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:22:53.1184613Z cpu  eval  hf_Bert_large                      
2025-09-07T09:23:22.2268311Z 
2025-09-07T09:23:22.4571556Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:23:22.6946940Z running benchmark:   3% 1/30 [00:00<00:06,  4.33it/s]
2025-09-07T09:23:22.9256393Z running benchmark:   7% 2/30 [00:00<00:06,  4.26it/s]
2025-09-07T09:23:23.1630061Z running benchmark:  10% 3/30 [00:00<00:06,  4.29it/s]
2025-09-07T09:23:23.3966068Z running benchmark:  13% 4/30 [00:00<00:06,  4.26it/s]
2025-09-07T09:23:23.6289147Z running benchmark:  17% 5/30 [00:01<00:05,  4.27it/s]
2025-09-07T09:23:23.8642730Z running benchmark:  20% 6/30 [00:01<00:05,  4.28it/s]
2025-09-07T09:23:24.1727777Z running benchmark:  23% 7/30 [00:01<00:05,  4.27it/s]
2025-09-07T09:23:24.4063833Z running benchmark:  27% 8/30 [00:01<00:05,  3.88it/s]
2025-09-07T09:23:24.6378510Z running benchmark:  30% 9/30 [00:02<00:05,  4.00it/s]
2025-09-07T09:23:24.8705298Z running benchmark:  33% 10/30 [00:02<00:04,  4.09it/s]
2025-09-07T09:23:25.1058843Z running benchmark:  37% 11/30 [00:02<00:04,  4.15it/s]
2025-09-07T09:23:25.3409072Z running benchmark:  40% 12/30 [00:02<00:04,  4.18it/s]
2025-09-07T09:23:25.5778482Z running benchmark:  43% 13/30 [00:03<00:04,  4.20it/s]
2025-09-07T09:23:25.8160516Z running benchmark:  47% 14/30 [00:03<00:03,  4.21it/s]
2025-09-07T09:23:26.0608086Z running benchmark:  50% 15/30 [00:03<00:03,  4.21it/s]
2025-09-07T09:23:26.2939088Z running benchmark:  53% 16/30 [00:03<00:03,  4.17it/s]
2025-09-07T09:23:26.5288016Z running benchmark:  57% 17/30 [00:04<00:03,  4.20it/s]
2025-09-07T09:23:26.7648953Z running benchmark:  60% 18/30 [00:04<00:02,  4.22it/s]
2025-09-07T09:23:27.0004510Z running benchmark:  63% 19/30 [00:04<00:02,  4.22it/s]
2025-09-07T09:23:27.2357433Z running benchmark:  67% 20/30 [00:04<00:02,  4.23it/s]
2025-09-07T09:23:27.4703325Z running benchmark:  70% 21/30 [00:05<00:02,  4.24it/s]
2025-09-07T09:23:27.7040339Z running benchmark:  73% 22/30 [00:05<00:01,  4.24it/s]
2025-09-07T09:23:27.9368971Z running benchmark:  77% 23/30 [00:05<00:01,  4.25it/s]
2025-09-07T09:23:28.2379419Z running benchmark:  80% 24/30 [00:05<00:01,  4.27it/s]
2025-09-07T09:23:28.4749888Z running benchmark:  83% 25/30 [00:06<00:01,  3.93it/s]
2025-09-07T09:23:28.7109604Z running benchmark:  87% 26/30 [00:06<00:00,  4.01it/s]
2025-09-07T09:23:28.9439256Z running benchmark:  90% 27/30 [00:06<00:00,  4.08it/s]
2025-09-07T09:23:29.1768556Z running benchmark:  93% 28/30 [00:06<00:00,  4.14it/s]
2025-09-07T09:23:29.4119215Z running benchmark:  97% 29/30 [00:06<00:00,  4.18it/s]
2025-09-07T09:23:29.4119596Z running benchmark: 100% 30/30 [00:07<00:00,  4.21it/s]
2025-09-07T09:23:29.4119911Z running benchmark: 100% 30/30 [00:07<00:00,  4.17it/s]
2025-09-07T09:23:29.5473756Z 1.210x
2025-09-07T09:23:29.5478066Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:23:32.5171308Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:23:32.5172487Z   import pynvml  # type: ignore[import]
2025-09-07T09:23:34.5278781Z 
2025-09-07T09:23:35.9370105Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T09:23:35.9371445Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T09:23:35.9372569Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T09:23:35.9373246Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T09:23:35.9374525Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T09:23:35.9375740Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T09:23:35.9376541Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T09:23:35.9377173Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T09:23:37.6115391Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T09:23:37.6116053Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T09:23:38.1921096Z 
2025-09-07T09:23:38.1923016Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:23:38.2126807Z cpu  eval  hf_BigBird                         
2025-09-07T09:24:12.1027182Z W0907 09:24:12.101865 197683 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] torch._dynamo hit config.recompile_limit (8)
2025-09-07T09:24:12.1028166Z W0907 09:24:12.101865 197683 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    function: 'forward' (/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/big_bird/modeling_big_bird.py:417)
2025-09-07T09:24:12.1030022Z W0907 09:24:12.101865 197683 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8]    last reason: 10/7: self.seed == 7                                           # context_layer, attention_probs = self.bigbird_block_sparse_attention(  # transformers/models/big_bird/modeling_big_bird.py:455 in forward (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
2025-09-07T09:24:12.1031906Z W0907 09:24:12.101865 197683 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-09-07T09:24:12.1032801Z W0907 09:24:12.101865 197683 site-packages/torch/_dynamo/convert_frame.py:1358] [10/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
2025-09-07T09:24:16.6955688Z 
2025-09-07T09:24:17.6647410Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:24:18.5643403Z running benchmark:   3% 1/30 [00:00<00:28,  1.03it/s]
2025-09-07T09:24:19.4441689Z running benchmark:   7% 2/30 [00:01<00:25,  1.08it/s]
2025-09-07T09:24:20.3340591Z running benchmark:  10% 3/30 [00:02<00:24,  1.10it/s]
2025-09-07T09:24:21.2868921Z running benchmark:  13% 4/30 [00:03<00:23,  1.11it/s]
2025-09-07T09:24:22.1682612Z running benchmark:  17% 5/30 [00:04<00:22,  1.09it/s]
2025-09-07T09:24:23.0571968Z running benchmark:  20% 6/30 [00:05<00:21,  1.10it/s]
2025-09-07T09:24:23.9402811Z running benchmark:  23% 7/30 [00:06<00:20,  1.11it/s]
2025-09-07T09:24:24.8236637Z running benchmark:  27% 8/30 [00:07<00:19,  1.12it/s]
2025-09-07T09:24:25.7797750Z running benchmark:  30% 9/30 [00:08<00:18,  1.12it/s]
2025-09-07T09:24:26.6640620Z running benchmark:  33% 10/30 [00:09<00:18,  1.10it/s]
2025-09-07T09:24:27.5530132Z running benchmark:  37% 11/30 [00:09<00:17,  1.11it/s]
2025-09-07T09:24:28.4417454Z running benchmark:  40% 12/30 [00:10<00:16,  1.11it/s]
2025-09-07T09:24:29.4020007Z running benchmark:  43% 13/30 [00:11<00:15,  1.12it/s]
2025-09-07T09:24:30.2840580Z running benchmark:  47% 14/30 [00:12<00:14,  1.09it/s]
2025-09-07T09:24:31.1739882Z running benchmark:  50% 15/30 [00:13<00:13,  1.10it/s]
2025-09-07T09:24:32.0630439Z running benchmark:  53% 16/30 [00:14<00:12,  1.11it/s]
2025-09-07T09:24:32.9531273Z running benchmark:  57% 17/30 [00:15<00:11,  1.11it/s]
2025-09-07T09:24:33.9136938Z running benchmark:  60% 18/30 [00:16<00:10,  1.12it/s]
2025-09-07T09:24:34.7957934Z running benchmark:  63% 19/30 [00:17<00:10,  1.09it/s]
2025-09-07T09:24:35.6807569Z running benchmark:  67% 20/30 [00:18<00:09,  1.11it/s]
2025-09-07T09:24:36.5674799Z running benchmark:  70% 21/30 [00:18<00:08,  1.11it/s]
2025-09-07T09:24:37.5192157Z running benchmark:  73% 22/30 [00:19<00:07,  1.12it/s]
2025-09-07T09:24:38.4028947Z running benchmark:  77% 23/30 [00:20<00:06,  1.10it/s]
2025-09-07T09:24:39.2901248Z running benchmark:  80% 24/30 [00:21<00:05,  1.11it/s]
2025-09-07T09:24:40.1731351Z running benchmark:  83% 25/30 [00:22<00:04,  1.11it/s]
2025-09-07T09:24:41.0612818Z running benchmark:  87% 26/30 [00:23<00:03,  1.12it/s]
2025-09-07T09:24:42.0224474Z running benchmark:  90% 27/30 [00:24<00:02,  1.12it/s]
2025-09-07T09:24:42.9043911Z running benchmark:  93% 28/30 [00:25<00:01,  1.10it/s]
2025-09-07T09:24:43.7889072Z running benchmark:  97% 29/30 [00:26<00:00,  1.11it/s]
2025-09-07T09:24:43.7889413Z running benchmark: 100% 30/30 [00:27<00:00,  1.11it/s]
2025-09-07T09:24:43.7889809Z running benchmark: 100% 30/30 [00:27<00:00,  1.11it/s]
2025-09-07T09:24:43.8937422Z 1.027x
2025-09-07T09:24:43.8942203Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:24:46.5838896Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:24:46.5840337Z   import pynvml  # type: ignore[import]
2025-09-07T09:24:48.6018406Z 
2025-09-07T09:24:51.0833216Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:24:51.0833802Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:24:51.0928151Z cpu  eval  hf_DistilBert                      
2025-09-07T09:25:03.3182602Z 
2025-09-07T09:25:03.4338338Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:25:03.5447695Z running benchmark:   7% 2/30 [00:00<00:01, 17.33it/s]
2025-09-07T09:25:03.6564669Z running benchmark:  13% 4/30 [00:00<00:01, 17.75it/s]
2025-09-07T09:25:03.7673631Z running benchmark:  20% 6/30 [00:00<00:01, 17.82it/s]
2025-09-07T09:25:03.8766786Z running benchmark:  27% 8/30 [00:00<00:01, 17.90it/s]
2025-09-07T09:25:03.9875965Z running benchmark:  33% 10/30 [00:00<00:01, 18.03it/s]
2025-09-07T09:25:04.0999690Z running benchmark:  40% 12/30 [00:00<00:00, 18.04it/s]
2025-09-07T09:25:04.2098204Z running benchmark:  47% 14/30 [00:00<00:00, 17.96it/s]
2025-09-07T09:25:04.3206249Z running benchmark:  53% 16/30 [00:00<00:00, 18.04it/s]
2025-09-07T09:25:04.4328852Z running benchmark:  60% 18/30 [00:01<00:00, 18.04it/s]
2025-09-07T09:25:04.5445278Z running benchmark:  67% 20/30 [00:01<00:00, 17.97it/s]
2025-09-07T09:25:04.6548665Z running benchmark:  73% 22/30 [00:01<00:00, 17.96it/s]
2025-09-07T09:25:04.7656231Z running benchmark:  80% 24/30 [00:01<00:00, 18.01it/s]
2025-09-07T09:25:04.8748896Z running benchmark:  87% 26/30 [00:01<00:00, 18.02it/s]
2025-09-07T09:25:04.9856170Z running benchmark:  93% 28/30 [00:01<00:00, 18.11it/s]
2025-09-07T09:25:04.9856875Z running benchmark: 100% 30/30 [00:01<00:00, 18.09it/s]
2025-09-07T09:25:04.9857545Z running benchmark: 100% 30/30 [00:01<00:00, 17.99it/s]
2025-09-07T09:25:05.0601595Z 1.232x
2025-09-07T09:25:05.0606698Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:25:07.4508228Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:25:07.4509146Z   import pynvml  # type: ignore[import]
2025-09-07T09:25:09.4742948Z 
2025-09-07T09:25:14.0589200Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:25:14.0589508Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:25:14.0747138Z cpu  eval  hf_GPT2                            
2025-09-07T09:25:31.3614298Z 
2025-09-07T09:25:31.5287578Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:25:31.6904851Z running benchmark:   7% 2/30 [00:00<00:02, 11.97it/s]
2025-09-07T09:25:31.8545903Z running benchmark:  13% 4/30 [00:00<00:02, 12.20it/s]
2025-09-07T09:25:32.0163376Z running benchmark:  20% 6/30 [00:00<00:01, 12.19it/s]
2025-09-07T09:25:32.1779746Z running benchmark:  27% 8/30 [00:00<00:01, 12.26it/s]
2025-09-07T09:25:32.3407563Z running benchmark:  33% 10/30 [00:00<00:01, 12.30it/s]
2025-09-07T09:25:32.5035839Z running benchmark:  40% 12/30 [00:00<00:01, 12.30it/s]
2025-09-07T09:25:32.6649968Z running benchmark:  47% 14/30 [00:01<00:01, 12.29it/s]
2025-09-07T09:25:32.8272898Z running benchmark:  53% 16/30 [00:01<00:01, 12.32it/s]
2025-09-07T09:25:32.9891132Z running benchmark:  60% 18/30 [00:01<00:00, 12.32it/s]
2025-09-07T09:25:33.1517346Z running benchmark:  67% 20/30 [00:01<00:00, 12.33it/s]
2025-09-07T09:25:33.3148533Z running benchmark:  73% 22/30 [00:01<00:00, 12.32it/s]
2025-09-07T09:25:33.4772785Z running benchmark:  80% 24/30 [00:01<00:00, 12.30it/s]
2025-09-07T09:25:33.6390468Z running benchmark:  87% 26/30 [00:02<00:00, 12.31it/s]
2025-09-07T09:25:33.8025599Z running benchmark:  93% 28/30 [00:02<00:00, 12.32it/s]
2025-09-07T09:25:33.8025995Z running benchmark: 100% 30/30 [00:02<00:00, 12.30it/s]
2025-09-07T09:25:33.8026305Z running benchmark: 100% 30/30 [00:02<00:00, 12.29it/s]
2025-09-07T09:25:33.8924286Z 1.405x
2025-09-07T09:25:33.8929109Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:25:36.5836062Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:25:36.5836944Z   import pynvml  # type: ignore[import]
2025-09-07T09:25:38.6039718Z 
2025-09-07T09:25:50.7243825Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:25:50.7244560Z loading model: 0it [00:12, ?it/s]
2025-09-07T09:25:50.7823957Z cpu  eval  hf_GPT2_large                      
2025-09-07T09:26:37.9268480Z 
2025-09-07T09:26:38.9024728Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:26:39.9349494Z running benchmark:   3% 1/30 [00:00<00:28,  1.03it/s]
2025-09-07T09:26:40.9044693Z running benchmark:   7% 2/30 [00:02<00:28,  1.01s/it]
2025-09-07T09:26:41.8703534Z running benchmark:  10% 3/30 [00:02<00:26,  1.01it/s]
2025-09-07T09:26:42.8344653Z running benchmark:  13% 4/30 [00:03<00:25,  1.02it/s]
2025-09-07T09:26:43.8409682Z running benchmark:  17% 5/30 [00:04<00:24,  1.03it/s]
2025-09-07T09:26:44.8090080Z running benchmark:  20% 6/30 [00:05<00:23,  1.01it/s]
2025-09-07T09:26:45.7820496Z running benchmark:  23% 7/30 [00:06<00:22,  1.02it/s]
2025-09-07T09:26:46.7462541Z running benchmark:  27% 8/30 [00:07<00:21,  1.02it/s]
2025-09-07T09:26:47.7069759Z running benchmark:  30% 9/30 [00:08<00:20,  1.03it/s]
2025-09-07T09:26:48.6733806Z running benchmark:  33% 10/30 [00:09<00:19,  1.03it/s]
2025-09-07T09:26:49.6379343Z running benchmark:  37% 11/30 [00:10<00:18,  1.03it/s]
2025-09-07T09:26:50.6021325Z running benchmark:  40% 12/30 [00:11<00:17,  1.03it/s]
2025-09-07T09:26:51.5648682Z running benchmark:  43% 13/30 [00:12<00:16,  1.03it/s]
2025-09-07T09:26:52.5306847Z running benchmark:  47% 14/30 [00:13<00:15,  1.04it/s]
2025-09-07T09:26:53.4989699Z running benchmark:  50% 15/30 [00:14<00:14,  1.04it/s]
2025-09-07T09:26:54.4671073Z running benchmark:  53% 16/30 [00:15<00:13,  1.03it/s]
2025-09-07T09:26:55.4330291Z running benchmark:  57% 17/30 [00:16<00:12,  1.03it/s]
2025-09-07T09:26:56.4000546Z running benchmark:  60% 18/30 [00:17<00:11,  1.03it/s]
2025-09-07T09:26:57.3682040Z running benchmark:  63% 19/30 [00:18<00:10,  1.03it/s]
2025-09-07T09:26:58.3324383Z running benchmark:  67% 20/30 [00:19<00:09,  1.03it/s]
2025-09-07T09:26:59.3018884Z running benchmark:  70% 21/30 [00:20<00:08,  1.03it/s]
2025-09-07T09:27:00.2681612Z running benchmark:  73% 22/30 [00:21<00:07,  1.03it/s]
2025-09-07T09:27:01.2409950Z running benchmark:  77% 23/30 [00:22<00:06,  1.03it/s]
2025-09-07T09:27:02.2081872Z running benchmark:  80% 24/30 [00:23<00:05,  1.03it/s]
2025-09-07T09:27:03.1704277Z running benchmark:  83% 25/30 [00:24<00:04,  1.03it/s]
2025-09-07T09:27:04.1301340Z running benchmark:  87% 26/30 [00:25<00:03,  1.03it/s]
2025-09-07T09:27:05.0919502Z running benchmark:  90% 27/30 [00:26<00:02,  1.04it/s]
2025-09-07T09:27:06.0553924Z running benchmark:  93% 28/30 [00:27<00:01,  1.04it/s]
2025-09-07T09:27:07.0221905Z running benchmark:  97% 29/30 [00:28<00:00,  1.04it/s]
2025-09-07T09:27:07.0222223Z running benchmark: 100% 30/30 [00:29<00:00,  1.04it/s]
2025-09-07T09:27:07.0222535Z running benchmark: 100% 30/30 [00:29<00:00,  1.03it/s]
2025-09-07T09:27:07.2584362Z 1.060x
2025-09-07T09:27:07.2589511Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:27:10.9468956Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:27:10.9470081Z   import pynvml  # type: ignore[import]
2025-09-07T09:27:12.9736967Z 
2025-09-07T09:27:16.6924803Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:27:16.6925624Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:27:16.7160255Z cpu  eval  hf_Longformer                      
2025-09-07T09:27:17.9141494Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T09:27:17.9142185Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T09:27:17.9142729Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] or:
2025-09-07T09:27:17.9143202Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T09:27:17.9144114Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] to include these operations in the captured graph.
2025-09-07T09:27:17.9144615Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:27:17.9145083Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] Graph break: from user code at:
2025-09-07T09:27:17.9145758Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_pass
2025-09-07T09:27:17.9146441Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     return mod(**inputs)
2025-09-07T09:27:17.9147222Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1703, in forward
2025-09-07T09:27:17.9148153Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     outputs = self.longformer(
2025-09-07T09:27:17.9148924Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1600, in forward
2025-09-07T09:27:17.9149683Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     encoder_outputs = self.encoder(
2025-09-07T09:27:17.9150443Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1244, in forward
2025-09-07T09:27:17.9151238Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2025-09-07T09:27:17.9151728Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:27:17.9152101Z W0907 09:27:17.913370 204281 site-packages/torch/_dynamo/variables/tensor.py:1048] [0/0] 
2025-09-07T09:28:36.0595155Z 
2025-09-07T09:28:36.3001150Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:28:36.5386594Z running benchmark:   3% 1/30 [00:00<00:06,  4.16it/s]
2025-09-07T09:28:36.7775328Z running benchmark:   7% 2/30 [00:00<00:06,  4.18it/s]
2025-09-07T09:28:37.0149518Z running benchmark:  10% 3/30 [00:00<00:06,  4.18it/s]
2025-09-07T09:28:37.2543688Z running benchmark:  13% 4/30 [00:00<00:06,  4.19it/s]
2025-09-07T09:28:37.4918297Z running benchmark:  17% 5/30 [00:01<00:05,  4.19it/s]
2025-09-07T09:28:37.7303734Z running benchmark:  20% 6/30 [00:01<00:05,  4.20it/s]
2025-09-07T09:28:37.9662322Z running benchmark:  23% 7/30 [00:01<00:05,  4.20it/s]
2025-09-07T09:28:38.2052628Z running benchmark:  27% 8/30 [00:01<00:05,  4.21it/s]
2025-09-07T09:28:38.4399356Z running benchmark:  30% 9/30 [00:02<00:04,  4.20it/s]
2025-09-07T09:28:38.6799800Z running benchmark:  33% 10/30 [00:02<00:04,  4.22it/s]
2025-09-07T09:28:38.9137376Z running benchmark:  37% 11/30 [00:02<00:04,  4.20it/s]
2025-09-07T09:28:39.1500902Z running benchmark:  40% 12/30 [00:02<00:04,  4.23it/s]
2025-09-07T09:28:39.3845641Z running benchmark:  43% 13/30 [00:03<00:04,  4.23it/s]
2025-09-07T09:28:39.6210516Z running benchmark:  47% 14/30 [00:03<00:03,  4.24it/s]
2025-09-07T09:28:39.8566451Z running benchmark:  50% 15/30 [00:03<00:03,  4.24it/s]
2025-09-07T09:28:40.0940590Z running benchmark:  53% 16/30 [00:03<00:03,  4.24it/s]
2025-09-07T09:28:40.3288962Z running benchmark:  57% 17/30 [00:04<00:03,  4.23it/s]
2025-09-07T09:28:40.5639113Z running benchmark:  60% 18/30 [00:04<00:02,  4.24it/s]
2025-09-07T09:28:40.7995905Z running benchmark:  63% 19/30 [00:04<00:02,  4.24it/s]
2025-09-07T09:28:41.0358276Z running benchmark:  67% 20/30 [00:04<00:02,  4.24it/s]
2025-09-07T09:28:41.2710417Z running benchmark:  70% 21/30 [00:04<00:02,  4.24it/s]
2025-09-07T09:28:41.5079660Z running benchmark:  73% 22/30 [00:05<00:01,  4.24it/s]
2025-09-07T09:28:41.7438994Z running benchmark:  77% 23/30 [00:05<00:01,  4.24it/s]
2025-09-07T09:28:41.9799642Z running benchmark:  80% 24/30 [00:05<00:01,  4.24it/s]
2025-09-07T09:28:42.2148887Z running benchmark:  83% 25/30 [00:05<00:01,  4.24it/s]
2025-09-07T09:28:42.4511881Z running benchmark:  87% 26/30 [00:06<00:00,  4.24it/s]
2025-09-07T09:28:42.6869479Z running benchmark:  90% 27/30 [00:06<00:00,  4.24it/s]
2025-09-07T09:28:42.9240484Z running benchmark:  93% 28/30 [00:06<00:00,  4.24it/s]
2025-09-07T09:28:43.1590390Z running benchmark:  97% 29/30 [00:06<00:00,  4.23it/s]
2025-09-07T09:28:43.1590687Z running benchmark: 100% 30/30 [00:07<00:00,  4.24it/s]
2025-09-07T09:28:43.1590928Z running benchmark: 100% 30/30 [00:07<00:00,  4.23it/s]
2025-09-07T09:28:43.2644272Z 1.408x
2025-09-07T09:28:43.2645004Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:28:47.1899955Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:28:47.1900847Z   import pynvml  # type: ignore[import]
2025-09-07T09:28:49.2055281Z 
2025-09-07T09:28:50.5477687Z loading model: 0it [00:00, ?it/s]Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T09:28:50.5479440Z WARNING:transformers.models.reformer.modeling_reformer:Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `ReformerDynamicCache` instead, e.g. `past_key_values=ReformerDynamicCache.from_legacy_cache(past_key_values)`.
2025-09-07T09:28:50.7845170Z 
2025-09-07T09:28:50.7845758Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:28:50.7885471Z cpu  eval  hf_Reformer                        
2025-09-07T09:28:59.2482056Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break from `Tensor.item()`, consider setting:
2025-09-07T09:28:59.2482776Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-09-07T09:28:59.2483311Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] or:
2025-09-07T09:28:59.2484161Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-09-07T09:28:59.2484765Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] to include these operations in the captured graph.
2025-09-07T09:28:59.2485263Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T09:28:59.2485735Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] Graph break: from user code at:
2025-09-07T09:28:59.2486591Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 2132, in torch_dynamo_resume_in_forward_at_2104
2025-09-07T09:28:59.2487552Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     embedding_output = self.embeddings(
2025-09-07T09:28:59.2488301Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 369, in forward
2025-09-07T09:28:59.2489102Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     position_embeddings = self.position_embeddings(position_ids)
2025-09-07T09:28:59.2489999Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]   File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/reformer/modeling_reformer.py", line 293, in forward
2025-09-07T09:28:59.2490767Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0]     max_position_id = position_ids.max().item()
2025-09-07T09:28:59.2491236Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T09:28:59.2491614Z W0907 09:28:59.247304 205729 site-packages/torch/_dynamo/variables/tensor.py:1048] [4/0] 
2025-09-07T09:29:16.1593593Z 
2025-09-07T09:29:16.5960008Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:29:17.0291709Z running benchmark:   3% 1/30 [00:00<00:12,  2.29it/s]
2025-09-07T09:29:17.4633337Z running benchmark:   7% 2/30 [00:00<00:12,  2.30it/s]
2025-09-07T09:29:17.8964601Z running benchmark:  10% 3/30 [00:01<00:11,  2.30it/s]
2025-09-07T09:29:18.3333261Z running benchmark:  13% 4/30 [00:01<00:11,  2.30it/s]
2025-09-07T09:29:18.7705429Z running benchmark:  17% 5/30 [00:02<00:10,  2.30it/s]
2025-09-07T09:29:19.2028612Z running benchmark:  20% 6/30 [00:02<00:10,  2.29it/s]
2025-09-07T09:29:19.6362769Z running benchmark:  23% 7/30 [00:03<00:09,  2.30it/s]
2025-09-07T09:29:20.0696449Z running benchmark:  27% 8/30 [00:03<00:09,  2.30it/s]
2025-09-07T09:29:20.5068597Z running benchmark:  30% 9/30 [00:03<00:09,  2.30it/s]
2025-09-07T09:29:20.9413529Z running benchmark:  33% 10/30 [00:04<00:08,  2.30it/s]
2025-09-07T09:29:21.3762323Z running benchmark:  37% 11/30 [00:04<00:08,  2.30it/s]
2025-09-07T09:29:21.8120389Z running benchmark:  40% 12/30 [00:05<00:07,  2.30it/s]
2025-09-07T09:29:22.2459627Z running benchmark:  43% 13/30 [00:05<00:07,  2.30it/s]
2025-09-07T09:29:22.6776772Z running benchmark:  47% 14/30 [00:06<00:06,  2.30it/s]
2025-09-07T09:29:23.1107235Z running benchmark:  50% 15/30 [00:06<00:06,  2.30it/s]
2025-09-07T09:29:23.5425819Z running benchmark:  53% 16/30 [00:06<00:06,  2.31it/s]
2025-09-07T09:29:23.9717050Z running benchmark:  57% 17/30 [00:07<00:05,  2.31it/s]
2025-09-07T09:29:24.4046325Z running benchmark:  60% 18/30 [00:07<00:05,  2.32it/s]
2025-09-07T09:29:24.8368395Z running benchmark:  63% 19/30 [00:08<00:04,  2.31it/s]
2025-09-07T09:29:25.2684910Z running benchmark:  67% 20/30 [00:08<00:04,  2.31it/s]
2025-09-07T09:29:25.7001557Z running benchmark:  70% 21/30 [00:09<00:03,  2.31it/s]
2025-09-07T09:29:26.1339534Z running benchmark:  73% 22/30 [00:09<00:03,  2.32it/s]
2025-09-07T09:29:26.5660980Z running benchmark:  77% 23/30 [00:09<00:03,  2.31it/s]
2025-09-07T09:29:26.9991801Z running benchmark:  80% 24/30 [00:10<00:02,  2.31it/s]
2025-09-07T09:29:27.4317485Z running benchmark:  83% 25/30 [00:10<00:02,  2.31it/s]
2025-09-07T09:29:27.8643460Z running benchmark:  87% 26/30 [00:11<00:01,  2.31it/s]
2025-09-07T09:29:28.2952575Z running benchmark:  90% 27/30 [00:11<00:01,  2.31it/s]
2025-09-07T09:29:28.7307847Z running benchmark:  93% 28/30 [00:12<00:00,  2.31it/s]
2025-09-07T09:29:29.1622643Z running benchmark:  97% 29/30 [00:12<00:00,  2.31it/s]
2025-09-07T09:29:29.1622971Z running benchmark: 100% 30/30 [00:13<00:00,  2.31it/s]
2025-09-07T09:29:29.1623595Z running benchmark: 100% 30/30 [00:13<00:00,  2.31it/s]
2025-09-07T09:29:29.2184343Z 1.094x
2025-09-07T09:29:29.2184725Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:29:31.6991361Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:29:31.6992267Z   import pynvml  # type: ignore[import]
2025-09-07T09:29:33.7287115Z 
2025-09-07T09:29:39.5291856Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:29:39.5292412Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:29:39.5545811Z cpu  eval  hf_Roberta_base                    
2025-09-07T09:29:57.3740826Z 
2025-09-07T09:29:57.5609034Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:29:57.7357639Z running benchmark:   3% 1/30 [00:00<00:05,  5.36it/s]
2025-09-07T09:29:57.9019412Z running benchmark:   7% 2/30 [00:00<00:05,  5.56it/s]
2025-09-07T09:29:58.0749397Z running benchmark:  10% 3/30 [00:00<00:04,  5.76it/s]
2025-09-07T09:29:58.2424520Z running benchmark:  13% 4/30 [00:00<00:04,  5.77it/s]
2025-09-07T09:29:58.4119815Z running benchmark:  17% 5/30 [00:00<00:04,  5.84it/s]
2025-09-07T09:29:58.5838545Z running benchmark:  20% 6/30 [00:01<00:04,  5.86it/s]
2025-09-07T09:29:58.7495984Z running benchmark:  23% 7/30 [00:01<00:03,  5.85it/s]
2025-09-07T09:29:58.9210306Z running benchmark:  27% 8/30 [00:01<00:03,  5.90it/s]
2025-09-07T09:29:59.0949146Z running benchmark:  30% 9/30 [00:01<00:03,  5.88it/s]
2025-09-07T09:29:59.3328179Z running benchmark:  33% 10/30 [00:01<00:03,  5.84it/s]
2025-09-07T09:29:59.5043107Z running benchmark:  37% 11/30 [00:01<00:03,  5.22it/s]
2025-09-07T09:29:59.6753864Z running benchmark:  40% 12/30 [00:02<00:03,  5.39it/s]
2025-09-07T09:29:59.8412191Z running benchmark:  43% 13/30 [00:02<00:03,  5.52it/s]
2025-09-07T09:30:00.0152968Z running benchmark:  47% 14/30 [00:02<00:02,  5.67it/s]
2025-09-07T09:30:00.1839460Z running benchmark:  50% 15/30 [00:02<00:02,  5.69it/s]
2025-09-07T09:30:00.3504731Z running benchmark:  53% 16/30 [00:02<00:02,  5.76it/s]
2025-09-07T09:30:00.5243347Z running benchmark:  57% 17/30 [00:02<00:02,  5.83it/s]
2025-09-07T09:30:00.6942035Z running benchmark:  60% 18/30 [00:03<00:02,  5.81it/s]
2025-09-07T09:30:00.8634189Z running benchmark:  63% 19/30 [00:03<00:01,  5.83it/s]
2025-09-07T09:30:01.0323589Z running benchmark:  67% 20/30 [00:03<00:01,  5.85it/s]
2025-09-07T09:30:01.2041056Z running benchmark:  70% 21/30 [00:03<00:01,  5.87it/s]
2025-09-07T09:30:01.3724430Z running benchmark:  73% 22/30 [00:03<00:01,  5.86it/s]
2025-09-07T09:30:01.5410764Z running benchmark:  77% 23/30 [00:03<00:01,  5.88it/s]
2025-09-07T09:30:01.7139508Z running benchmark:  80% 24/30 [00:04<00:01,  5.90it/s]
2025-09-07T09:30:01.8798006Z running benchmark:  83% 25/30 [00:04<00:00,  5.86it/s]
2025-09-07T09:30:02.0501375Z running benchmark:  87% 26/30 [00:04<00:00,  5.91it/s]
2025-09-07T09:30:02.2247912Z running benchmark:  90% 27/30 [00:04<00:00,  5.90it/s]
2025-09-07T09:30:02.3922890Z running benchmark:  93% 28/30 [00:04<00:00,  5.85it/s]
2025-09-07T09:30:02.5650371Z running benchmark:  97% 29/30 [00:05<00:00,  5.88it/s]
2025-09-07T09:30:02.5651120Z running benchmark: 100% 30/30 [00:05<00:00,  5.85it/s]
2025-09-07T09:30:02.5651390Z running benchmark: 100% 30/30 [00:05<00:00,  5.78it/s]
2025-09-07T09:30:02.6982092Z 1.109x
2025-09-07T09:30:02.6986420Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:30:05.3386650Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:30:05.3387496Z   import pynvml  # type: ignore[import]
2025-09-07T09:30:07.3611658Z 
2025-09-07T09:30:09.7681971Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:30:09.7682561Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:30:09.7843850Z cpu  eval  hf_T5                              
2025-09-07T09:30:32.5402792Z 
2025-09-07T09:30:32.6820180Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:30:32.8218552Z running benchmark:   3% 1/30 [00:00<00:04,  7.06it/s]
2025-09-07T09:30:32.9596512Z running benchmark:   7% 2/30 [00:00<00:03,  7.12it/s]
2025-09-07T09:30:33.0991809Z running benchmark:  10% 3/30 [00:00<00:03,  7.18it/s]
2025-09-07T09:30:33.2351967Z running benchmark:  13% 4/30 [00:00<00:03,  7.17it/s]
2025-09-07T09:30:33.3743123Z running benchmark:  17% 5/30 [00:00<00:03,  7.24it/s]
2025-09-07T09:30:33.5104398Z running benchmark:  20% 6/30 [00:00<00:03,  7.22it/s]
2025-09-07T09:30:33.6500577Z running benchmark:  23% 7/30 [00:00<00:03,  7.26it/s]
2025-09-07T09:30:33.7868112Z running benchmark:  27% 8/30 [00:01<00:03,  7.23it/s]
2025-09-07T09:30:33.9270124Z running benchmark:  30% 9/30 [00:01<00:02,  7.26it/s]
2025-09-07T09:30:34.0634784Z running benchmark:  33% 10/30 [00:01<00:02,  7.22it/s]
2025-09-07T09:30:34.2035664Z running benchmark:  37% 11/30 [00:01<00:02,  7.25it/s]
2025-09-07T09:30:34.3407499Z running benchmark:  40% 12/30 [00:01<00:02,  7.22it/s]
2025-09-07T09:30:34.4808812Z running benchmark:  43% 13/30 [00:01<00:02,  7.24it/s]
2025-09-07T09:30:34.6171092Z running benchmark:  47% 14/30 [00:01<00:02,  7.21it/s]
2025-09-07T09:30:34.7564271Z running benchmark:  50% 15/30 [00:02<00:02,  7.25it/s]
2025-09-07T09:30:34.8922456Z running benchmark:  53% 16/30 [00:02<00:01,  7.23it/s]
2025-09-07T09:30:35.0320616Z running benchmark:  57% 17/30 [00:02<00:01,  7.27it/s]
2025-09-07T09:30:35.1690685Z running benchmark:  60% 18/30 [00:02<00:01,  7.23it/s]
2025-09-07T09:30:35.3100791Z running benchmark:  63% 19/30 [00:02<00:01,  7.25it/s]
2025-09-07T09:30:35.4468260Z running benchmark:  67% 20/30 [00:02<00:01,  7.20it/s]
2025-09-07T09:30:35.5865763Z running benchmark:  70% 21/30 [00:02<00:01,  7.24it/s]
2025-09-07T09:30:35.7237072Z running benchmark:  73% 22/30 [00:03<00:01,  7.21it/s]
2025-09-07T09:30:35.8635629Z running benchmark:  77% 23/30 [00:03<00:00,  7.24it/s]
2025-09-07T09:30:36.0724128Z running benchmark:  80% 24/30 [00:03<00:00,  7.21it/s]
2025-09-07T09:30:36.2124661Z running benchmark:  83% 25/30 [00:03<00:00,  6.26it/s]
2025-09-07T09:30:36.3489386Z running benchmark:  87% 26/30 [00:03<00:00,  6.50it/s]
2025-09-07T09:30:36.4888409Z running benchmark:  90% 27/30 [00:03<00:00,  6.73it/s]
2025-09-07T09:30:36.6258837Z running benchmark:  93% 28/30 [00:03<00:00,  6.85it/s]
2025-09-07T09:30:36.7662667Z running benchmark:  97% 29/30 [00:04<00:00,  6.98it/s]
2025-09-07T09:30:36.7663026Z running benchmark: 100% 30/30 [00:04<00:00,  7.02it/s]
2025-09-07T09:30:36.7663512Z running benchmark: 100% 30/30 [00:04<00:00,  7.10it/s]
2025-09-07T09:30:36.8469917Z 1.983x
2025-09-07T09:30:36.8474440Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:30:39.5305223Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:30:39.5306385Z   import pynvml  # type: ignore[import]
2025-09-07T09:30:41.5531573Z 
2025-09-07T09:30:46.7835337Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:30:46.7835655Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:30:46.8247140Z cpu  eval  hf_T5_base                         
2025-09-07T09:31:37.4714811Z 
2025-09-07T09:31:39.0802459Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:31:40.6760984Z running benchmark:   3% 1/30 [00:01<00:46,  1.61s/it]
2025-09-07T09:31:42.3583047Z running benchmark:   7% 2/30 [00:03<00:44,  1.60s/it]
2025-09-07T09:31:43.9668237Z running benchmark:  10% 3/30 [00:04<00:44,  1.64s/it]
2025-09-07T09:31:45.6357274Z running benchmark:  13% 4/30 [00:06<00:42,  1.63s/it]
2025-09-07T09:31:47.2336744Z running benchmark:  17% 5/30 [00:08<00:41,  1.64s/it]
2025-09-07T09:31:48.8404676Z running benchmark:  20% 6/30 [00:09<00:39,  1.63s/it]
2025-09-07T09:31:50.5014191Z running benchmark:  23% 7/30 [00:11<00:37,  1.62s/it]
2025-09-07T09:31:52.1047788Z running benchmark:  27% 8/30 [00:13<00:35,  1.63s/it]
2025-09-07T09:31:53.7637196Z running benchmark:  30% 9/30 [00:14<00:34,  1.62s/it]
2025-09-07T09:31:55.3644983Z running benchmark:  33% 10/30 [00:16<00:32,  1.63s/it]
2025-09-07T09:31:56.9695907Z running benchmark:  37% 11/30 [00:17<00:30,  1.62s/it]
2025-09-07T09:31:58.6593419Z running benchmark:  40% 12/30 [00:19<00:29,  1.62s/it]
2025-09-07T09:32:00.2586954Z running benchmark:  43% 13/30 [00:21<00:27,  1.64s/it]
2025-09-07T09:32:01.9325206Z running benchmark:  47% 14/30 [00:22<00:26,  1.63s/it]
2025-09-07T09:32:03.5278017Z running benchmark:  50% 15/30 [00:24<00:24,  1.64s/it]
2025-09-07T09:32:05.1258365Z running benchmark:  53% 16/30 [00:26<00:22,  1.63s/it]
2025-09-07T09:32:06.8168640Z running benchmark:  57% 17/30 [00:27<00:21,  1.62s/it]
2025-09-07T09:32:08.4259048Z running benchmark:  60% 18/30 [00:29<00:19,  1.64s/it]
2025-09-07T09:32:10.1107024Z running benchmark:  63% 19/30 [00:30<00:17,  1.63s/it]
2025-09-07T09:32:11.7563481Z running benchmark:  67% 20/30 [00:32<00:16,  1.65s/it]
2025-09-07T09:32:13.3540489Z running benchmark:  70% 21/30 [00:34<00:14,  1.65s/it]
2025-09-07T09:32:15.0229079Z running benchmark:  73% 22/30 [00:35<00:13,  1.63s/it]
2025-09-07T09:32:16.6204098Z running benchmark:  77% 23/30 [00:37<00:11,  1.64s/it]
2025-09-07T09:32:18.2900660Z running benchmark:  80% 24/30 [00:39<00:09,  1.63s/it]
2025-09-07T09:32:19.8971726Z running benchmark:  83% 25/30 [00:40<00:08,  1.64s/it]
2025-09-07T09:32:21.4929868Z running benchmark:  87% 26/30 [00:42<00:06,  1.63s/it]
2025-09-07T09:32:23.1549699Z running benchmark:  90% 27/30 [00:44<00:04,  1.62s/it]
2025-09-07T09:32:24.7685119Z running benchmark:  93% 28/30 [00:45<00:03,  1.63s/it]
2025-09-07T09:32:26.4509376Z running benchmark:  97% 29/30 [00:47<00:01,  1.63s/it]
2025-09-07T09:32:26.4509689Z running benchmark: 100% 30/30 [00:48<00:00,  1.64s/it]
2025-09-07T09:32:26.4509933Z running benchmark: 100% 30/30 [00:48<00:00,  1.63s/it]
2025-09-07T09:32:26.8531706Z 1.437x
2025-09-07T09:32:26.8535872Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:32:30.8118452Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:32:30.8119378Z   import pynvml  # type: ignore[import]
2025-09-07T09:32:32.8357289Z 
2025-09-07T09:32:42.0784010Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:32:42.0784574Z loading model: 0it [00:09, ?it/s]
2025-09-07T09:32:42.1646412Z cpu  eval  hf_T5_large                        
2025-09-07T09:34:21.2612447Z 
2025-09-07T09:34:22.1260916Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:34:22.9978516Z running benchmark:   3% 1/30 [00:00<00:25,  1.16it/s]
2025-09-07T09:34:23.8741487Z running benchmark:   7% 2/30 [00:01<00:24,  1.15it/s]
2025-09-07T09:34:24.8176202Z running benchmark:  10% 3/30 [00:02<00:23,  1.15it/s]
2025-09-07T09:34:25.6877839Z running benchmark:  13% 4/30 [00:03<00:23,  1.11it/s]
2025-09-07T09:34:26.5593790Z running benchmark:  17% 5/30 [00:04<00:22,  1.12it/s]
2025-09-07T09:34:27.4305978Z running benchmark:  20% 6/30 [00:05<00:21,  1.13it/s]
2025-09-07T09:34:28.3752189Z running benchmark:  23% 7/30 [00:06<00:20,  1.14it/s]
2025-09-07T09:34:29.2448399Z running benchmark:  27% 8/30 [00:07<00:19,  1.11it/s]
2025-09-07T09:34:30.1188361Z running benchmark:  30% 9/30 [00:07<00:18,  1.12it/s]
2025-09-07T09:34:30.9899381Z running benchmark:  33% 10/30 [00:08<00:17,  1.13it/s]
2025-09-07T09:34:31.8657351Z running benchmark:  37% 11/30 [00:09<00:16,  1.13it/s]
2025-09-07T09:34:32.8131722Z running benchmark:  40% 12/30 [00:10<00:15,  1.14it/s]
2025-09-07T09:34:33.6909177Z running benchmark:  43% 13/30 [00:11<00:15,  1.11it/s]
2025-09-07T09:34:34.5595733Z running benchmark:  47% 14/30 [00:12<00:14,  1.12it/s]
2025-09-07T09:34:35.4393782Z running benchmark:  50% 15/30 [00:13<00:13,  1.13it/s]
2025-09-07T09:34:36.3848448Z running benchmark:  53% 16/30 [00:14<00:12,  1.13it/s]
2025-09-07T09:34:37.2658101Z running benchmark:  57% 17/30 [00:15<00:11,  1.11it/s]
2025-09-07T09:34:38.1355985Z running benchmark:  60% 18/30 [00:16<00:10,  1.12it/s]
2025-09-07T09:34:39.0201781Z running benchmark:  63% 19/30 [00:16<00:09,  1.13it/s]
2025-09-07T09:34:39.8935769Z running benchmark:  67% 20/30 [00:17<00:08,  1.13it/s]
2025-09-07T09:34:40.8449522Z running benchmark:  70% 21/30 [00:18<00:07,  1.13it/s]
2025-09-07T09:34:41.7306657Z running benchmark:  73% 22/30 [00:19<00:07,  1.11it/s]
2025-09-07T09:34:42.6450986Z running benchmark:  77% 23/30 [00:20<00:06,  1.11it/s]
2025-09-07T09:34:43.5483982Z running benchmark:  80% 24/30 [00:21<00:05,  1.11it/s]
2025-09-07T09:34:44.5301367Z running benchmark:  83% 25/30 [00:22<00:04,  1.11it/s]
2025-09-07T09:34:45.3939038Z running benchmark:  87% 26/30 [00:23<00:03,  1.08it/s]
2025-09-07T09:34:46.2709483Z running benchmark:  90% 27/30 [00:24<00:02,  1.10it/s]
2025-09-07T09:34:47.1344797Z running benchmark:  93% 28/30 [00:25<00:01,  1.11it/s]
2025-09-07T09:34:48.0037377Z running benchmark:  97% 29/30 [00:25<00:00,  1.13it/s]
2025-09-07T09:34:48.0037891Z running benchmark: 100% 30/30 [00:26<00:00,  1.13it/s]
2025-09-07T09:34:48.0038183Z running benchmark: 100% 30/30 [00:26<00:00,  1.12it/s]
2025-09-07T09:34:48.2532110Z 1.501x
2025-09-07T09:34:48.2535783Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:34:52.4430894Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:34:52.4431764Z   import pynvml  # type: ignore[import]
2025-09-07T09:34:54.4580365Z 
2025-09-07T09:34:59.1567135Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:34:59.1567474Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:34:59.1913614Z cpu  eval  hf_distil_whisper                  
2025-09-07T09:35:35.8152979Z 
2025-09-07T09:35:36.4387079Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:35:37.0600518Z running benchmark:   3% 1/30 [00:00<00:18,  1.61it/s]
2025-09-07T09:35:37.7476444Z running benchmark:   7% 2/30 [00:01<00:17,  1.61it/s]
2025-09-07T09:35:38.3677772Z running benchmark:  10% 3/30 [00:01<00:17,  1.53it/s]
2025-09-07T09:35:38.9846286Z running benchmark:  13% 4/30 [00:02<00:16,  1.56it/s]
2025-09-07T09:35:39.5853145Z running benchmark:  17% 5/30 [00:03<00:15,  1.58it/s]
2025-09-07T09:35:40.2086876Z running benchmark:  20% 6/30 [00:03<00:14,  1.61it/s]
2025-09-07T09:35:40.8338183Z running benchmark:  23% 7/30 [00:04<00:14,  1.61it/s]
2025-09-07T09:35:41.5259002Z running benchmark:  27% 8/30 [00:05<00:13,  1.61it/s]
2025-09-07T09:35:42.1368322Z running benchmark:  30% 9/30 [00:05<00:13,  1.55it/s]
2025-09-07T09:35:42.7476369Z running benchmark:  33% 10/30 [00:06<00:12,  1.58it/s]
2025-09-07T09:35:43.3618819Z running benchmark:  37% 11/30 [00:06<00:11,  1.59it/s]
2025-09-07T09:35:43.9838823Z running benchmark:  40% 12/30 [00:07<00:11,  1.61it/s]
2025-09-07T09:35:44.6051347Z running benchmark:  43% 13/30 [00:08<00:10,  1.61it/s]
2025-09-07T09:35:45.2222004Z running benchmark:  47% 14/30 [00:08<00:09,  1.61it/s]
2025-09-07T09:35:45.9064991Z running benchmark:  50% 15/30 [00:09<00:09,  1.61it/s]
2025-09-07T09:35:46.5323442Z running benchmark:  53% 16/30 [00:10<00:08,  1.56it/s]
2025-09-07T09:35:47.1498675Z running benchmark:  57% 17/30 [00:10<00:08,  1.57it/s]
2025-09-07T09:35:47.7676895Z running benchmark:  60% 18/30 [00:11<00:07,  1.59it/s]
2025-09-07T09:35:48.3890551Z running benchmark:  63% 19/30 [00:11<00:06,  1.60it/s]
2025-09-07T09:35:48.9970880Z running benchmark:  67% 20/30 [00:12<00:06,  1.60it/s]
2025-09-07T09:35:49.6292224Z running benchmark:  70% 21/30 [00:13<00:05,  1.61it/s]
2025-09-07T09:35:50.2406351Z running benchmark:  73% 22/30 [00:13<00:04,  1.60it/s]
2025-09-07T09:35:50.8682712Z running benchmark:  77% 23/30 [00:14<00:04,  1.61it/s]
2025-09-07T09:35:51.4924735Z running benchmark:  80% 24/30 [00:15<00:03,  1.61it/s]
2025-09-07T09:35:52.1125258Z running benchmark:  83% 25/30 [00:15<00:03,  1.61it/s]
2025-09-07T09:35:52.7322721Z running benchmark:  87% 26/30 [00:16<00:02,  1.61it/s]
2025-09-07T09:35:53.3544948Z running benchmark:  90% 27/30 [00:16<00:01,  1.61it/s]
2025-09-07T09:35:53.9794365Z running benchmark:  93% 28/30 [00:17<00:01,  1.61it/s]
2025-09-07T09:35:54.6088202Z running benchmark:  97% 29/30 [00:18<00:00,  1.61it/s]
2025-09-07T09:35:54.6088702Z running benchmark: 100% 30/30 [00:18<00:00,  1.60it/s]
2025-09-07T09:35:54.6089012Z running benchmark: 100% 30/30 [00:18<00:00,  1.60it/s]
2025-09-07T09:35:54.7615133Z 1.195x
2025-09-07T09:35:54.7619548Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:35:57.7510303Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:35:57.7511196Z   import pynvml  # type: ignore[import]
2025-09-07T09:35:59.7715096Z 
2025-09-07T09:35:59.9674500Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:35:59.9674788Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:35:59.9675415Z cpu  eval  lennard_jones                      
2025-09-07T09:36:07.2751267Z 
2025-09-07T09:36:07.3492383Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:36:07.3493120Z running benchmark: 100% 30/30 [00:00<00:00, 406.50it/s]
2025-09-07T09:36:07.4003014Z 0.844x
2025-09-07T09:36:07.4008359Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:36:08.8957991Z speedup             gmean=1.23x mean=1.267x
2025-09-07T09:36:08.8958666Z abs_latency         gmean=46.92x mean=134.539x
2025-09-07T09:36:08.8959400Z compilation_latency mean=25.788 seconds
2025-09-07T09:36:08.8960032Z compression_ratio   mean=0.908x
2025-09-07T09:36:08.8963082Z eager_peak_mem      gmean=205.58x mean=331.508x
2025-09-07T09:36:08.8965003Z dynamo_peak_mem     gmean=226.87x mean=348.971x
2025-09-07T09:36:08.8967178Z calls_captured      gmean=273.07x mean=611.609x
2025-09-07T09:36:08.8969265Z unique_graphs       gmean=1.66x mean=3.043x
2025-09-07T09:36:08.8971306Z graph_breaks        gmean=0.00x mean=2.174x
2025-09-07T09:36:08.8973423Z unique_graph_breaks gmean=0.00x mean=0.565x
2025-09-07T09:36:08.8975581Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T09:36:08.8977683Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T09:36:08.8979827Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T09:36:09.4693610Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *freezing_cudagraphs-true* ]]
2025-09-07T09:36:09.4694761Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *freeze_autotune_cudagraphs-true* ]]
2025-09-07T09:36:09.4695507Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *aotinductor-true* ]]
2025-09-07T09:36:09.4695951Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2025-09-07T09:36:09.4696156Z + [[ performance == \a\c\c\u\r\a\c\y ]]
2025-09-07T09:36:09.4697160Z + taskset -c 0-94 python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cpu --total-partitions 4 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv
2025-09-07T09:36:09.8629146Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:36:09.8630059Z   import pynvml  # type: ignore[import]
2025-09-07T09:36:12.3232486Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:36:12.3233389Z   import pynvml  # type: ignore[import]
2025-09-07T09:36:14.3427345Z 
2025-09-07T09:36:20.2010868Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:36:20.2011469Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:36:20.2210294Z cpu  eval  dlrm                               
2025-09-07T09:36:40.7947872Z 
2025-09-07T09:36:40.9433561Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:36:41.0790348Z running benchmark:   7% 2/30 [00:00<00:02, 13.49it/s]
2025-09-07T09:36:41.2146865Z running benchmark:  13% 4/30 [00:00<00:01, 14.20it/s]
2025-09-07T09:36:41.3499707Z running benchmark:  20% 6/30 [00:00<00:01, 14.44it/s]
2025-09-07T09:36:41.4861299Z running benchmark:  27% 8/30 [00:00<00:01, 14.57it/s]
2025-09-07T09:36:41.6216555Z running benchmark:  33% 10/30 [00:00<00:01, 14.62it/s]
2025-09-07T09:36:41.7570184Z running benchmark:  40% 12/30 [00:00<00:01, 14.66it/s]
2025-09-07T09:36:41.8930648Z running benchmark:  47% 14/30 [00:00<00:01, 14.70it/s]
2025-09-07T09:36:42.0286926Z running benchmark:  53% 16/30 [00:01<00:00, 14.70it/s]
2025-09-07T09:36:42.1651205Z running benchmark:  60% 18/30 [00:01<00:00, 14.72it/s]
2025-09-07T09:36:42.3007155Z running benchmark:  67% 20/30 [00:01<00:00, 14.70it/s]
2025-09-07T09:36:42.4368031Z running benchmark:  73% 22/30 [00:01<00:00, 14.71it/s]
2025-09-07T09:36:42.5912803Z running benchmark:  80% 24/30 [00:01<00:00, 14.70it/s]
2025-09-07T09:36:42.7280279Z running benchmark:  87% 26/30 [00:01<00:00, 14.13it/s]
2025-09-07T09:36:42.8629952Z running benchmark:  93% 28/30 [00:01<00:00, 14.27it/s]
2025-09-07T09:36:42.8630629Z running benchmark: 100% 30/30 [00:02<00:00, 14.43it/s]
2025-09-07T09:36:42.8631806Z running benchmark: 100% 30/30 [00:02<00:00, 14.51it/s]
2025-09-07T09:36:42.9880340Z 0.971x
2025-09-07T09:36:42.9882856Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:36:44.9574370Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:36:44.9575245Z   import pynvml  # type: ignore[import]
2025-09-07T09:36:46.9704409Z 
2025-09-07T09:36:47.3784447Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:36:47.3784744Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:36:47.3810498Z cpu  eval  functorch_dp_cifar10               
2025-09-07T09:37:01.1171345Z 
2025-09-07T09:37:01.2204772Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:37:01.3401508Z running benchmark:  13% 4/30 [00:00<00:00, 38.90it/s]
2025-09-07T09:37:01.4593832Z running benchmark:  33% 10/30 [00:00<00:00, 45.88it/s]
2025-09-07T09:37:01.5787136Z running benchmark:  53% 16/30 [00:00<00:00, 47.96it/s]
2025-09-07T09:37:01.6984789Z running benchmark:  73% 22/30 [00:00<00:00, 48.89it/s]
2025-09-07T09:37:01.7388300Z running benchmark:  93% 28/30 [00:00<00:00, 49.33it/s]
2025-09-07T09:37:01.7388637Z running benchmark: 100% 30/30 [00:00<00:00, 48.27it/s]
2025-09-07T09:37:01.7922837Z 0.758x
2025-09-07T09:37:01.7925522Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:37:03.5189206Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:37:03.5190184Z   import pynvml  # type: ignore[import]
2025-09-07T09:37:05.5359724Z 
2025-09-07T09:37:05.7162879Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:37:05.7163379Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:37:05.7168887Z cpu  eval  functorch_maml_omniglot            
2025-09-07T09:37:16.3827377Z 
2025-09-07T09:37:16.4858685Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:37:16.4859020Z running benchmark: 100% 30/30 [00:00<00:00, 291.80it/s]
2025-09-07T09:37:16.4859740Z running benchmark: 100% 30/30 [00:00<00:00, 291.27it/s]
2025-09-07T09:37:16.5355173Z 1.071x
2025-09-07T09:37:16.5358348Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:37:18.0729175Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:37:18.0730067Z   import pynvml  # type: ignore[import]
2025-09-07T09:37:20.0949797Z 
2025-09-07T09:37:21.6291012Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:37:21.6291321Z loading model: 0it [00:01, ?it/s]
2025-09-07T09:37:21.6320295Z cpu  eval  hf_Albert                          
2025-09-07T09:37:38.0144888Z 
2025-09-07T09:37:38.1263760Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:37:38.3135219Z running benchmark:   3% 1/30 [00:00<00:03,  8.96it/s]
2025-09-07T09:37:38.5020974Z running benchmark:  10% 3/30 [00:00<00:02, 10.18it/s]
2025-09-07T09:37:38.6882788Z running benchmark:  17% 5/30 [00:00<00:02, 10.40it/s]
2025-09-07T09:37:38.8729301Z running benchmark:  23% 7/30 [00:00<00:02, 10.54it/s]
2025-09-07T09:37:39.0569933Z running benchmark:  30% 9/30 [00:00<00:01, 10.65it/s]
2025-09-07T09:37:39.2409632Z running benchmark:  37% 11/30 [00:01<00:01, 10.72it/s]
2025-09-07T09:37:39.4254461Z running benchmark:  43% 13/30 [00:01<00:01, 10.77it/s]
2025-09-07T09:37:39.6101403Z running benchmark:  50% 15/30 [00:01<00:01, 10.79it/s]
2025-09-07T09:37:39.7953158Z running benchmark:  57% 17/30 [00:01<00:01, 10.80it/s]
2025-09-07T09:37:39.9814369Z running benchmark:  63% 19/30 [00:01<00:01, 10.80it/s]
2025-09-07T09:37:40.1660854Z running benchmark:  70% 21/30 [00:01<00:00, 10.79it/s]
2025-09-07T09:37:40.3515261Z running benchmark:  77% 23/30 [00:02<00:00, 10.80it/s]
2025-09-07T09:37:40.5375040Z running benchmark:  83% 25/30 [00:02<00:00, 10.79it/s]
2025-09-07T09:37:40.7215795Z running benchmark:  90% 27/30 [00:02<00:00, 10.78it/s]
2025-09-07T09:37:40.8144281Z running benchmark:  97% 29/30 [00:02<00:00, 10.81it/s]
2025-09-07T09:37:40.8144621Z running benchmark: 100% 30/30 [00:02<00:00, 10.72it/s]
2025-09-07T09:37:40.8771483Z 1.384x
2025-09-07T09:37:40.8775421Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:37:42.8013711Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:37:42.8014929Z   import pynvml  # type: ignore[import]
2025-09-07T09:37:44.8142692Z 
2025-09-07T09:37:48.6832399Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:37:48.6833032Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:37:48.7061102Z cpu  eval  hf_Bart                            
2025-09-07T09:38:11.0971243Z 
2025-09-07T09:38:11.2437219Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:38:11.3784489Z running benchmark:   3% 1/30 [00:00<00:04,  6.84it/s]
2025-09-07T09:38:11.5126207Z running benchmark:   7% 2/30 [00:00<00:03,  7.17it/s]
2025-09-07T09:38:11.6600389Z running benchmark:  10% 3/30 [00:00<00:03,  7.30it/s]
2025-09-07T09:38:11.8023856Z running benchmark:  13% 4/30 [00:00<00:03,  7.09it/s]
2025-09-07T09:38:11.9356599Z running benchmark:  17% 5/30 [00:00<00:03,  7.06it/s]
2025-09-07T09:38:12.0683302Z running benchmark:  20% 6/30 [00:00<00:03,  7.21it/s]
2025-09-07T09:38:12.2017540Z running benchmark:  23% 7/30 [00:00<00:03,  7.31it/s]
2025-09-07T09:38:12.3355951Z running benchmark:  27% 8/30 [00:01<00:02,  7.37it/s]
2025-09-07T09:38:12.4752705Z running benchmark:  30% 9/30 [00:01<00:02,  7.40it/s]
2025-09-07T09:38:12.6080726Z running benchmark:  33% 10/30 [00:01<00:02,  7.32it/s]
2025-09-07T09:38:12.7421323Z running benchmark:  37% 11/30 [00:01<00:02,  7.39it/s]
2025-09-07T09:38:12.8792668Z running benchmark:  40% 12/30 [00:01<00:02,  7.41it/s]
2025-09-07T09:38:13.0134119Z running benchmark:  43% 13/30 [00:01<00:02,  7.37it/s]
2025-09-07T09:38:13.1473906Z running benchmark:  47% 14/30 [00:01<00:02,  7.40it/s]
2025-09-07T09:38:13.2806029Z running benchmark:  50% 15/30 [00:02<00:02,  7.42it/s]
2025-09-07T09:38:13.4136012Z running benchmark:  53% 16/30 [00:02<00:01,  7.45it/s]
2025-09-07T09:38:13.5467106Z running benchmark:  57% 17/30 [00:02<00:01,  7.47it/s]
2025-09-07T09:38:13.6806375Z running benchmark:  60% 18/30 [00:02<00:01,  7.48it/s]
2025-09-07T09:38:13.8133419Z running benchmark:  63% 19/30 [00:02<00:01,  7.48it/s]
2025-09-07T09:38:13.9475094Z running benchmark:  67% 20/30 [00:02<00:01,  7.50it/s]
2025-09-07T09:38:14.0814589Z running benchmark:  70% 21/30 [00:02<00:01,  7.48it/s]
2025-09-07T09:38:14.2152082Z running benchmark:  73% 22/30 [00:02<00:01,  7.48it/s]
2025-09-07T09:38:14.3481262Z running benchmark:  77% 23/30 [00:03<00:00,  7.48it/s]
2025-09-07T09:38:14.4800504Z running benchmark:  80% 24/30 [00:03<00:00,  7.49it/s]
2025-09-07T09:38:14.6127953Z running benchmark:  83% 25/30 [00:03<00:00,  7.52it/s]
2025-09-07T09:38:14.7456106Z running benchmark:  87% 26/30 [00:03<00:00,  7.52it/s]
2025-09-07T09:38:14.8783162Z running benchmark:  90% 27/30 [00:03<00:00,  7.52it/s]
2025-09-07T09:38:15.0101591Z running benchmark:  93% 28/30 [00:03<00:00,  7.53it/s]
2025-09-07T09:38:15.1415953Z running benchmark:  97% 29/30 [00:03<00:00,  7.55it/s]
2025-09-07T09:38:15.1416316Z running benchmark: 100% 30/30 [00:04<00:00,  7.56it/s]
2025-09-07T09:38:15.1416595Z running benchmark: 100% 30/30 [00:04<00:00,  7.42it/s]
2025-09-07T09:38:15.2655507Z 1.257x
2025-09-07T09:38:15.2659443Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:38:17.5910197Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:38:17.5911122Z   import pynvml  # type: ignore[import]
2025-09-07T09:38:19.6094004Z 
2025-09-07T09:38:22.4758999Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:38:22.4759330Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:38:22.4925981Z cpu  eval  hf_Bert                            
2025-09-07T09:38:41.3680585Z 
2025-09-07T09:38:41.4814057Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:38:41.5833173Z running benchmark:   3% 1/30 [00:00<00:03,  8.84it/s]
2025-09-07T09:38:41.6869908Z running benchmark:   7% 2/30 [00:00<00:02,  9.39it/s]
2025-09-07T09:38:41.7910031Z running benchmark:  10% 3/30 [00:00<00:02,  9.50it/s]
2025-09-07T09:38:41.8943499Z running benchmark:  13% 4/30 [00:00<00:02,  9.55it/s]
2025-09-07T09:38:41.9968749Z running benchmark:  17% 5/30 [00:00<00:02,  9.59it/s]
2025-09-07T09:38:42.0982623Z running benchmark:  20% 6/30 [00:00<00:02,  9.65it/s]
2025-09-07T09:38:42.2011194Z running benchmark:  23% 7/30 [00:00<00:02,  9.72it/s]
2025-09-07T09:38:42.3038479Z running benchmark:  27% 8/30 [00:00<00:02,  9.72it/s]
2025-09-07T09:38:42.4058090Z running benchmark:  30% 9/30 [00:00<00:02,  9.72it/s]
2025-09-07T09:38:42.5065502Z running benchmark:  33% 10/30 [00:01<00:02,  9.75it/s]
2025-09-07T09:38:42.6104957Z running benchmark:  37% 11/30 [00:01<00:01,  9.80it/s]
2025-09-07T09:38:42.7131578Z running benchmark:  40% 12/30 [00:01<00:01,  9.75it/s]
2025-09-07T09:38:42.8159093Z running benchmark:  43% 13/30 [00:01<00:01,  9.75it/s]
2025-09-07T09:38:42.9181817Z running benchmark:  47% 14/30 [00:01<00:01,  9.74it/s]
2025-09-07T09:38:43.0228522Z running benchmark:  50% 15/30 [00:01<00:01,  9.75it/s]
2025-09-07T09:38:43.1247790Z running benchmark:  53% 16/30 [00:01<00:01,  9.69it/s]
2025-09-07T09:38:43.2284935Z running benchmark:  57% 17/30 [00:01<00:01,  9.73it/s]
2025-09-07T09:38:43.3301784Z running benchmark:  60% 18/30 [00:01<00:01,  9.70it/s]
2025-09-07T09:38:43.4328213Z running benchmark:  63% 19/30 [00:01<00:01,  9.74it/s]
2025-09-07T09:38:43.5353845Z running benchmark:  67% 20/30 [00:02<00:01,  9.74it/s]
2025-09-07T09:38:43.6378096Z running benchmark:  70% 21/30 [00:02<00:00,  9.75it/s]
2025-09-07T09:38:43.7390334Z running benchmark:  73% 22/30 [00:02<00:00,  9.75it/s]
2025-09-07T09:38:43.8424398Z running benchmark:  77% 23/30 [00:02<00:00,  9.79it/s]
2025-09-07T09:38:43.9457358Z running benchmark:  80% 24/30 [00:02<00:00,  9.75it/s]
2025-09-07T09:38:44.0482897Z running benchmark:  83% 25/30 [00:02<00:00,  9.73it/s]
2025-09-07T09:38:44.1507996Z running benchmark:  87% 26/30 [00:02<00:00,  9.74it/s]
2025-09-07T09:38:44.2539472Z running benchmark:  90% 27/30 [00:02<00:00,  9.74it/s]
2025-09-07T09:38:44.3564535Z running benchmark:  93% 28/30 [00:02<00:00,  9.73it/s]
2025-09-07T09:38:44.4592322Z running benchmark:  97% 29/30 [00:02<00:00,  9.74it/s]
2025-09-07T09:38:44.4592674Z running benchmark: 100% 30/30 [00:03<00:00,  9.74it/s]
2025-09-07T09:38:44.4593382Z running benchmark: 100% 30/30 [00:03<00:00,  9.71it/s]
2025-09-07T09:38:44.5588274Z 1.219x
2025-09-07T09:38:44.5592582Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:38:46.7242131Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:38:46.7245050Z   import pynvml  # type: ignore[import]
2025-09-07T09:38:48.7494817Z 
2025-09-07T09:38:53.3588667Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:38:53.3589077Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:38:53.3921184Z cpu  eval  hf_Bert_large                      
2025-09-07T09:39:21.7998621Z 
2025-09-07T09:39:22.1292413Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:39:22.3619778Z running benchmark:   3% 1/30 [00:00<00:09,  3.04it/s]
2025-09-07T09:39:22.5904310Z running benchmark:   7% 2/30 [00:00<00:07,  3.67it/s]
2025-09-07T09:39:22.8201645Z running benchmark:  10% 3/30 [00:00<00:06,  3.96it/s]
2025-09-07T09:39:23.0480615Z running benchmark:  13% 4/30 [00:01<00:06,  4.11it/s]
2025-09-07T09:39:23.2792673Z running benchmark:  17% 5/30 [00:01<00:05,  4.21it/s]
2025-09-07T09:39:23.5091595Z running benchmark:  20% 6/30 [00:01<00:05,  4.25it/s]
2025-09-07T09:39:23.7392788Z running benchmark:  23% 7/30 [00:01<00:05,  4.28it/s]
2025-09-07T09:39:23.9680417Z running benchmark:  27% 8/30 [00:01<00:05,  4.30it/s]
2025-09-07T09:39:24.1991985Z running benchmark:  30% 9/30 [00:02<00:04,  4.32it/s]
2025-09-07T09:39:24.4299187Z running benchmark:  33% 10/30 [00:02<00:04,  4.32it/s]
2025-09-07T09:39:24.6610918Z running benchmark:  37% 11/30 [00:02<00:04,  4.33it/s]
2025-09-07T09:39:24.8908713Z running benchmark:  40% 12/30 [00:02<00:04,  4.33it/s]
2025-09-07T09:39:25.1214054Z running benchmark:  43% 13/30 [00:03<00:03,  4.33it/s]
2025-09-07T09:39:25.3501865Z running benchmark:  47% 14/30 [00:03<00:03,  4.34it/s]
2025-09-07T09:39:25.5779187Z running benchmark:  50% 15/30 [00:03<00:03,  4.35it/s]
2025-09-07T09:39:25.8058435Z running benchmark:  53% 16/30 [00:03<00:03,  4.36it/s]
2025-09-07T09:39:26.0372816Z running benchmark:  57% 17/30 [00:04<00:02,  4.37it/s]
2025-09-07T09:39:26.2660487Z running benchmark:  60% 18/30 [00:04<00:02,  4.35it/s]
2025-09-07T09:39:26.4931656Z running benchmark:  63% 19/30 [00:04<00:02,  4.36it/s]
2025-09-07T09:39:26.7227932Z running benchmark:  67% 20/30 [00:04<00:02,  4.37it/s]
2025-09-07T09:39:26.9547005Z running benchmark:  70% 21/30 [00:04<00:02,  4.37it/s]
2025-09-07T09:39:27.1869033Z running benchmark:  73% 22/30 [00:05<00:01,  4.35it/s]
2025-09-07T09:39:27.4170714Z running benchmark:  77% 23/30 [00:05<00:01,  4.34it/s]
2025-09-07T09:39:27.6461168Z running benchmark:  80% 24/30 [00:05<00:01,  4.34it/s]
2025-09-07T09:39:27.8761906Z running benchmark:  83% 25/30 [00:05<00:01,  4.35it/s]
2025-09-07T09:39:28.1059907Z running benchmark:  87% 26/30 [00:06<00:00,  4.35it/s]
2025-09-07T09:39:28.3368986Z running benchmark:  90% 27/30 [00:06<00:00,  4.35it/s]
2025-09-07T09:39:28.5689159Z running benchmark:  93% 28/30 [00:06<00:00,  4.34it/s]
2025-09-07T09:39:28.7972334Z running benchmark:  97% 29/30 [00:06<00:00,  4.33it/s]
2025-09-07T09:39:28.7974069Z running benchmark: 100% 30/30 [00:06<00:00,  4.35it/s]
2025-09-07T09:39:28.7974556Z running benchmark: 100% 30/30 [00:06<00:00,  4.29it/s]
2025-09-07T09:39:28.9520661Z 1.237x
2025-09-07T09:39:28.9524419Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:39:31.5812662Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:39:31.5813796Z   import pynvml  # type: ignore[import]
2025-09-07T09:39:33.6044508Z 
2025-09-07T09:39:35.0139865Z loading model: 0it [00:00, ?it/s]BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T09:39:35.0141214Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T09:39:35.0142043Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T09:39:35.0142715Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T09:39:35.0143982Z WARNING:transformers.modeling_utils:BigBirdForMaskedLM has generative capabilities, as `prepare_inputs_for_generation` is explicitly defined. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
2025-09-07T09:39:35.0145202Z   - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
2025-09-07T09:39:35.0146110Z   - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
2025-09-07T09:39:35.0146737Z   - If you are not the owner of the model architecture class, please contact the model code owner to update it.
2025-09-07T09:39:36.6913193Z Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T09:39:36.6913876Z WARNING:transformers.models.big_bird.modeling_big_bird:Input ids are automatically padded from 819 to 832 to be a multiple of `config.block_size`: 64
2025-09-07T09:39:37.2430449Z 
2025-09-07T09:39:37.2432041Z loading model: 0it [00:03, ?it/s]
2025-09-07T09:39:37.2652077Z cpu  eval  hf_BigBird                         
2025-09-07T09:40:42.2271301Z 
2025-09-07T09:40:43.1319052Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:40:43.9519720Z running benchmark:   3% 1/30 [00:00<00:26,  1.11it/s]
2025-09-07T09:40:44.7738314Z running benchmark:   7% 2/30 [00:01<00:23,  1.17it/s]
2025-09-07T09:40:45.5968996Z running benchmark:  10% 3/30 [00:02<00:22,  1.19it/s]
2025-09-07T09:40:46.4197819Z running benchmark:  13% 4/30 [00:03<00:21,  1.20it/s]
2025-09-07T09:40:47.3098074Z running benchmark:  17% 5/30 [00:04<00:20,  1.21it/s]
2025-09-07T09:40:48.1326397Z running benchmark:  20% 6/30 [00:05<00:20,  1.18it/s]
2025-09-07T09:40:48.9590652Z running benchmark:  23% 7/30 [00:05<00:19,  1.19it/s]
2025-09-07T09:40:49.7797221Z running benchmark:  27% 8/30 [00:06<00:18,  1.20it/s]
2025-09-07T09:40:50.6048979Z running benchmark:  30% 9/30 [00:07<00:17,  1.20it/s]
2025-09-07T09:40:51.4985498Z running benchmark:  33% 10/30 [00:08<00:16,  1.21it/s]
2025-09-07T09:40:52.3198992Z running benchmark:  37% 11/30 [00:09<00:16,  1.18it/s]
2025-09-07T09:40:53.1450757Z running benchmark:  40% 12/30 [00:10<00:15,  1.19it/s]
2025-09-07T09:40:53.9644106Z running benchmark:  43% 13/30 [00:10<00:14,  1.20it/s]
2025-09-07T09:40:54.7847720Z running benchmark:  47% 14/30 [00:11<00:13,  1.20it/s]
2025-09-07T09:40:55.6739323Z running benchmark:  50% 15/30 [00:12<00:12,  1.21it/s]
2025-09-07T09:40:56.4961138Z running benchmark:  53% 16/30 [00:13<00:11,  1.18it/s]
2025-09-07T09:40:57.3197176Z running benchmark:  57% 17/30 [00:14<00:10,  1.19it/s]
2025-09-07T09:40:58.1447844Z running benchmark:  60% 18/30 [00:15<00:10,  1.20it/s]
2025-09-07T09:40:58.9698917Z running benchmark:  63% 19/30 [00:15<00:09,  1.20it/s]
2025-09-07T09:40:59.8191598Z running benchmark:  67% 20/30 [00:16<00:08,  1.21it/s]
2025-09-07T09:41:00.6389363Z running benchmark:  70% 21/30 [00:17<00:07,  1.20it/s]
2025-09-07T09:41:01.4598873Z running benchmark:  73% 22/30 [00:18<00:06,  1.20it/s]
2025-09-07T09:41:02.2838425Z running benchmark:  77% 23/30 [00:19<00:05,  1.21it/s]
2025-09-07T09:41:03.1066357Z running benchmark:  80% 24/30 [00:20<00:04,  1.21it/s]
2025-09-07T09:41:03.9259252Z running benchmark:  83% 25/30 [00:20<00:04,  1.21it/s]
2025-09-07T09:41:04.7494081Z running benchmark:  87% 26/30 [00:21<00:03,  1.21it/s]
2025-09-07T09:41:05.5730770Z running benchmark:  90% 27/30 [00:22<00:02,  1.21it/s]
2025-09-07T09:41:06.3953672Z running benchmark:  93% 28/30 [00:23<00:01,  1.21it/s]
2025-09-07T09:41:07.2175025Z running benchmark:  97% 29/30 [00:24<00:00,  1.21it/s]
2025-09-07T09:41:07.2175355Z running benchmark: 100% 30/30 [00:24<00:00,  1.22it/s]
2025-09-07T09:41:07.2175638Z running benchmark: 100% 30/30 [00:24<00:00,  1.20it/s]
2025-09-07T09:41:07.3236500Z 1.227x
2025-09-07T09:41:07.3240746Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:41:10.8790348Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:41:10.8791938Z   import pynvml  # type: ignore[import]
2025-09-07T09:41:12.8971177Z 
2025-09-07T09:41:15.3822361Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:41:15.3823127Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:41:15.3916324Z cpu  eval  hf_DistilBert                      
2025-09-07T09:41:30.4237998Z 
2025-09-07T09:41:30.5531226Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:41:30.6631547Z running benchmark:   7% 2/30 [00:00<00:01, 15.49it/s]
2025-09-07T09:41:30.7757683Z running benchmark:  13% 4/30 [00:00<00:01, 16.96it/s]
2025-09-07T09:41:30.8867999Z running benchmark:  20% 6/30 [00:00<00:01, 17.32it/s]
2025-09-07T09:41:30.9985196Z running benchmark:  27% 8/30 [00:00<00:01, 17.58it/s]
2025-09-07T09:41:31.1109873Z running benchmark:  33% 10/30 [00:00<00:01, 17.70it/s]
2025-09-07T09:41:31.2222302Z running benchmark:  40% 12/30 [00:00<00:01, 17.74it/s]
2025-09-07T09:41:31.3322786Z running benchmark:  47% 14/30 [00:00<00:00, 17.80it/s]
2025-09-07T09:41:31.4449066Z running benchmark:  53% 16/30 [00:00<00:00, 17.92it/s]
2025-09-07T09:41:31.5563802Z running benchmark:  60% 18/30 [00:01<00:00, 17.87it/s]
2025-09-07T09:41:31.6684855Z running benchmark:  67% 20/30 [00:01<00:00, 17.89it/s]
2025-09-07T09:41:31.7828046Z running benchmark:  73% 22/30 [00:01<00:00, 17.88it/s]
2025-09-07T09:41:31.8946301Z running benchmark:  80% 24/30 [00:01<00:00, 17.76it/s]
2025-09-07T09:41:32.0050627Z running benchmark:  87% 26/30 [00:01<00:00, 17.80it/s]
2025-09-07T09:41:32.1162953Z running benchmark:  93% 28/30 [00:01<00:00, 17.89it/s]
2025-09-07T09:41:32.1165930Z running benchmark: 100% 30/30 [00:01<00:00, 17.92it/s]
2025-09-07T09:41:32.1166335Z running benchmark: 100% 30/30 [00:01<00:00, 17.72it/s]
2025-09-07T09:41:32.1969577Z 1.169x
2025-09-07T09:41:32.1974068Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:41:34.1514053Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:41:34.1514975Z   import pynvml  # type: ignore[import]
2025-09-07T09:41:36.1597418Z 
2025-09-07T09:41:40.6676059Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:41:40.6677636Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:41:40.6839686Z cpu  eval  hf_GPT2                            
2025-09-07T09:42:00.6775291Z 
2025-09-07T09:42:00.8500439Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:42:01.0075396Z running benchmark:   7% 2/30 [00:00<00:02, 11.61it/s]
2025-09-07T09:42:01.1636905Z running benchmark:  13% 4/30 [00:00<00:02, 12.23it/s]
2025-09-07T09:42:01.3207830Z running benchmark:  20% 6/30 [00:00<00:01, 12.49it/s]
2025-09-07T09:42:01.4774384Z running benchmark:  27% 8/30 [00:00<00:01, 12.58it/s]
2025-09-07T09:42:01.6348259Z running benchmark:  33% 10/30 [00:00<00:01, 12.65it/s]
2025-09-07T09:42:01.7927349Z running benchmark:  40% 12/30 [00:00<00:01, 12.67it/s]
2025-09-07T09:42:01.9506188Z running benchmark:  47% 14/30 [00:01<00:01, 12.67it/s]
2025-09-07T09:42:02.1069688Z running benchmark:  53% 16/30 [00:01<00:01, 12.67it/s]
2025-09-07T09:42:02.2634889Z running benchmark:  60% 18/30 [00:01<00:00, 12.71it/s]
2025-09-07T09:42:02.4275834Z running benchmark:  67% 20/30 [00:01<00:00, 12.73it/s]
2025-09-07T09:42:02.5878305Z running benchmark:  73% 22/30 [00:01<00:00, 12.56it/s]
2025-09-07T09:42:02.7445218Z running benchmark:  80% 24/30 [00:01<00:00, 12.53it/s]
2025-09-07T09:42:02.9017630Z running benchmark:  87% 26/30 [00:02<00:00, 12.60it/s]
2025-09-07T09:42:03.0594464Z running benchmark:  93% 28/30 [00:02<00:00, 12.64it/s]
2025-09-07T09:42:03.0594985Z running benchmark: 100% 30/30 [00:02<00:00, 12.65it/s]
2025-09-07T09:42:03.0595293Z running benchmark: 100% 30/30 [00:02<00:00, 12.60it/s]
2025-09-07T09:42:03.1643227Z 1.374x
2025-09-07T09:42:03.1646538Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:42:05.4875186Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:42:05.4876064Z   import pynvml  # type: ignore[import]
2025-09-07T09:42:07.5098008Z 
2025-09-07T09:42:19.5382452Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:42:19.5383759Z loading model: 0it [00:12, ?it/s]
2025-09-07T09:42:19.5951561Z cpu  eval  hf_GPT2_large                      
2025-09-07T09:43:07.1124506Z 
2025-09-07T09:43:08.1395812Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:43:09.1830539Z running benchmark:   3% 1/30 [00:01<00:29,  1.03s/it]
2025-09-07T09:43:10.2112397Z running benchmark:   7% 2/30 [00:02<00:29,  1.04s/it]
2025-09-07T09:43:11.2481415Z running benchmark:  10% 3/30 [00:03<00:27,  1.03s/it]
2025-09-07T09:43:12.2752809Z running benchmark:  13% 4/30 [00:04<00:26,  1.03s/it]
2025-09-07T09:43:13.3112432Z running benchmark:  17% 5/30 [00:05<00:25,  1.03s/it]
2025-09-07T09:43:14.3078812Z running benchmark:  20% 6/30 [00:06<00:24,  1.03s/it]
2025-09-07T09:43:15.3277425Z running benchmark:  23% 7/30 [00:07<00:23,  1.02s/it]
2025-09-07T09:43:16.3393026Z running benchmark:  27% 8/30 [00:08<00:22,  1.02s/it]
2025-09-07T09:43:17.3585371Z running benchmark:  30% 9/30 [00:09<00:21,  1.02s/it]
2025-09-07T09:43:18.3659527Z running benchmark:  33% 10/30 [00:10<00:20,  1.02s/it]
2025-09-07T09:43:19.4093905Z running benchmark:  37% 11/30 [00:11<00:19,  1.01s/it]
2025-09-07T09:43:20.4159752Z running benchmark:  40% 12/30 [00:12<00:18,  1.02s/it]
2025-09-07T09:43:21.4322461Z running benchmark:  43% 13/30 [00:13<00:17,  1.02s/it]
2025-09-07T09:43:22.4460650Z running benchmark:  47% 14/30 [00:14<00:16,  1.02s/it]
2025-09-07T09:43:23.4662515Z running benchmark:  50% 15/30 [00:15<00:15,  1.02s/it]
2025-09-07T09:43:24.4901703Z running benchmark:  53% 16/30 [00:16<00:14,  1.02s/it]
2025-09-07T09:43:25.5121958Z running benchmark:  57% 17/30 [00:17<00:13,  1.02s/it]
2025-09-07T09:43:26.6038604Z running benchmark:  60% 18/30 [00:18<00:12,  1.02s/it]
2025-09-07T09:43:27.6454068Z running benchmark:  63% 19/30 [00:19<00:11,  1.04s/it]
2025-09-07T09:43:28.6698482Z running benchmark:  67% 20/30 [00:20<00:10,  1.04s/it]
2025-09-07T09:43:29.6935980Z running benchmark:  70% 21/30 [00:21<00:09,  1.04s/it]
2025-09-07T09:43:30.7614815Z running benchmark:  73% 22/30 [00:22<00:08,  1.03s/it]
2025-09-07T09:43:31.8020172Z running benchmark:  77% 23/30 [00:23<00:07,  1.04s/it]
2025-09-07T09:43:32.8386706Z running benchmark:  80% 24/30 [00:24<00:06,  1.04s/it]
2025-09-07T09:43:33.8865384Z running benchmark:  83% 25/30 [00:25<00:05,  1.04s/it]
2025-09-07T09:43:34.9916218Z running benchmark:  87% 26/30 [00:26<00:04,  1.04s/it]
2025-09-07T09:43:36.0654428Z running benchmark:  90% 27/30 [00:27<00:03,  1.06s/it]
2025-09-07T09:43:37.0975400Z running benchmark:  93% 28/30 [00:28<00:02,  1.07s/it]
2025-09-07T09:43:38.1314583Z running benchmark:  97% 29/30 [00:29<00:01,  1.06s/it]
2025-09-07T09:43:38.1314898Z running benchmark: 100% 30/30 [00:31<00:00,  1.05s/it]
2025-09-07T09:43:38.1315158Z running benchmark: 100% 30/30 [00:31<00:00,  1.03s/it]
2025-09-07T09:43:38.4461744Z 0.935x
2025-09-07T09:43:38.4468252Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:43:42.1556754Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:43:42.1557655Z   import pynvml  # type: ignore[import]
2025-09-07T09:43:44.1740137Z 
2025-09-07T09:43:49.9701455Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:43:49.9702131Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:43:49.9945884Z cpu  eval  hf_Roberta_base                    
2025-09-07T09:44:11.6223742Z 
2025-09-07T09:44:11.8191587Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:44:11.9953286Z running benchmark:   3% 1/30 [00:00<00:05,  5.09it/s]
2025-09-07T09:44:12.1661316Z running benchmark:   7% 2/30 [00:00<00:05,  5.42it/s]
2025-09-07T09:44:12.3354107Z running benchmark:  10% 3/30 [00:00<00:04,  5.61it/s]
2025-09-07T09:44:12.5019794Z running benchmark:  13% 4/30 [00:00<00:04,  5.72it/s]
2025-09-07T09:44:12.6755093Z running benchmark:  17% 5/30 [00:00<00:04,  5.82it/s]
2025-09-07T09:44:12.8449877Z running benchmark:  20% 6/30 [00:01<00:04,  5.80it/s]
2025-09-07T09:44:13.0117804Z running benchmark:  23% 7/30 [00:01<00:03,  5.83it/s]
2025-09-07T09:44:13.1856032Z running benchmark:  27% 8/30 [00:01<00:03,  5.88it/s]
2025-09-07T09:44:13.3552566Z running benchmark:  30% 9/30 [00:01<00:03,  5.84it/s]
2025-09-07T09:44:13.5185909Z running benchmark:  33% 10/30 [00:01<00:03,  5.86it/s]
2025-09-07T09:44:13.6955217Z running benchmark:  37% 11/30 [00:01<00:03,  5.94it/s]
2025-09-07T09:44:13.8658819Z running benchmark:  40% 12/30 [00:02<00:03,  5.85it/s]
2025-09-07T09:44:14.0271909Z running benchmark:  43% 13/30 [00:02<00:02,  5.85it/s]
2025-09-07T09:44:14.1953875Z running benchmark:  47% 14/30 [00:02<00:02,  5.95it/s]
2025-09-07T09:44:14.3653306Z running benchmark:  50% 15/30 [00:02<00:02,  5.95it/s]
2025-09-07T09:44:14.5311639Z running benchmark:  53% 16/30 [00:02<00:02,  5.93it/s]
2025-09-07T09:44:14.7055805Z running benchmark:  57% 17/30 [00:02<00:02,  5.96it/s]
2025-09-07T09:44:14.8744256Z running benchmark:  60% 18/30 [00:03<00:02,  5.89it/s]
2025-09-07T09:44:15.0410789Z running benchmark:  63% 19/30 [00:03<00:01,  5.90it/s]
2025-09-07T09:44:15.2153212Z running benchmark:  67% 20/30 [00:03<00:01,  5.93it/s]
2025-09-07T09:44:15.3844069Z running benchmark:  70% 21/30 [00:03<00:01,  5.87it/s]
2025-09-07T09:44:15.5485778Z running benchmark:  73% 22/30 [00:03<00:01,  5.88it/s]
2025-09-07T09:44:15.7260219Z running benchmark:  77% 23/30 [00:03<00:01,  5.95it/s]
2025-09-07T09:44:15.8944935Z running benchmark:  80% 24/30 [00:04<00:01,  5.85it/s]
2025-09-07T09:44:16.0614292Z running benchmark:  83% 25/30 [00:04<00:00,  5.87it/s]
2025-09-07T09:44:16.2354861Z running benchmark:  87% 26/30 [00:04<00:00,  5.91it/s]
2025-09-07T09:44:16.4037119Z running benchmark:  90% 27/30 [00:04<00:00,  5.86it/s]
2025-09-07T09:44:16.5713615Z running benchmark:  93% 28/30 [00:04<00:00,  5.88it/s]
2025-09-07T09:44:16.7370667Z running benchmark:  97% 29/30 [00:04<00:00,  5.91it/s]
2025-09-07T09:44:16.7370978Z running benchmark: 100% 30/30 [00:05<00:00,  5.95it/s]
2025-09-07T09:44:16.7371230Z running benchmark: 100% 30/30 [00:05<00:00,  5.87it/s]
2025-09-07T09:44:16.8874128Z 1.107x
2025-09-07T09:44:16.8878765Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:44:19.2513752Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:44:19.2514665Z   import pynvml  # type: ignore[import]
2025-09-07T09:44:21.2730340Z 
2025-09-07T09:44:23.6762052Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:44:23.6762346Z loading model: 0it [00:02, ?it/s]
2025-09-07T09:44:23.6919107Z cpu  eval  hf_T5                              
2025-09-07T09:44:46.4746639Z 
2025-09-07T09:44:46.6218325Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:44:46.7580251Z running benchmark:   3% 1/30 [00:00<00:04,  6.80it/s]
2025-09-07T09:44:46.8941221Z running benchmark:   7% 2/30 [00:00<00:03,  7.11it/s]
2025-09-07T09:44:47.0283885Z running benchmark:  10% 3/30 [00:00<00:03,  7.21it/s]
2025-09-07T09:44:47.1634044Z running benchmark:  13% 4/30 [00:00<00:03,  7.31it/s]
2025-09-07T09:44:47.2983348Z running benchmark:  17% 5/30 [00:00<00:03,  7.34it/s]
2025-09-07T09:44:47.4742218Z running benchmark:  20% 6/30 [00:00<00:03,  7.37it/s]
2025-09-07T09:44:47.6102942Z running benchmark:  23% 7/30 [00:00<00:03,  6.72it/s]
2025-09-07T09:44:47.7462182Z running benchmark:  27% 8/30 [00:01<00:03,  6.91it/s]
2025-09-07T09:44:47.8818261Z running benchmark:  30% 9/30 [00:01<00:02,  7.04it/s]
2025-09-07T09:44:48.0178576Z running benchmark:  33% 10/30 [00:01<00:02,  7.14it/s]
2025-09-07T09:44:48.1534105Z running benchmark:  37% 11/30 [00:01<00:02,  7.20it/s]
2025-09-07T09:44:48.2883383Z running benchmark:  40% 12/30 [00:01<00:02,  7.26it/s]
2025-09-07T09:44:48.4240237Z running benchmark:  43% 13/30 [00:01<00:02,  7.30it/s]
2025-09-07T09:44:48.5627282Z running benchmark:  47% 14/30 [00:01<00:02,  7.32it/s]
2025-09-07T09:44:48.6982018Z running benchmark:  50% 15/30 [00:02<00:02,  7.29it/s]
2025-09-07T09:44:48.8343574Z running benchmark:  53% 16/30 [00:02<00:01,  7.32it/s]
2025-09-07T09:44:48.9708104Z running benchmark:  57% 17/30 [00:02<00:01,  7.32it/s]
2025-09-07T09:44:49.1065105Z running benchmark:  60% 18/30 [00:02<00:01,  7.33it/s]
2025-09-07T09:44:49.2416386Z running benchmark:  63% 19/30 [00:02<00:01,  7.34it/s]
2025-09-07T09:44:49.3767692Z running benchmark:  67% 20/30 [00:02<00:01,  7.36it/s]
2025-09-07T09:44:49.5115124Z running benchmark:  70% 21/30 [00:02<00:01,  7.37it/s]
2025-09-07T09:44:49.6480628Z running benchmark:  73% 22/30 [00:03<00:01,  7.39it/s]
2025-09-07T09:44:49.7832707Z running benchmark:  77% 23/30 [00:03<00:00,  7.37it/s]
2025-09-07T09:44:49.9192190Z running benchmark:  80% 24/30 [00:03<00:00,  7.38it/s]
2025-09-07T09:44:50.0552736Z running benchmark:  83% 25/30 [00:03<00:00,  7.37it/s]
2025-09-07T09:44:50.1904477Z running benchmark:  87% 26/30 [00:03<00:00,  7.36it/s]
2025-09-07T09:44:50.3253201Z running benchmark:  90% 27/30 [00:03<00:00,  7.37it/s]
2025-09-07T09:44:50.4613220Z running benchmark:  93% 28/30 [00:03<00:00,  7.39it/s]
2025-09-07T09:44:50.5967226Z running benchmark:  97% 29/30 [00:03<00:00,  7.38it/s]
2025-09-07T09:44:50.5967570Z running benchmark: 100% 30/30 [00:04<00:00,  7.38it/s]
2025-09-07T09:44:50.5967912Z running benchmark: 100% 30/30 [00:04<00:00,  7.28it/s]
2025-09-07T09:44:50.6765051Z 2.046x
2025-09-07T09:44:50.6768708Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:44:52.9351400Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:44:52.9352267Z   import pynvml  # type: ignore[import]
2025-09-07T09:44:54.9503028Z 
2025-09-07T09:45:00.1748373Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:45:00.1748818Z loading model: 0it [00:05, ?it/s]
2025-09-07T09:45:00.2173909Z cpu  eval  hf_T5_base                         
2025-09-07T09:45:44.4132578Z 
2025-09-07T09:45:46.0561594Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:45:47.6210132Z running benchmark:   3% 1/30 [00:01<00:47,  1.64s/it]
2025-09-07T09:45:49.1979006Z running benchmark:   7% 2/30 [00:03<00:44,  1.60s/it]
2025-09-07T09:45:50.7591044Z running benchmark:  10% 3/30 [00:04<00:42,  1.59s/it]
2025-09-07T09:45:52.3363730Z running benchmark:  13% 4/30 [00:06<00:41,  1.58s/it]
2025-09-07T09:45:53.8956530Z running benchmark:  17% 5/30 [00:07<00:39,  1.58s/it]
2025-09-07T09:45:55.4636174Z running benchmark:  20% 6/30 [00:09<00:37,  1.57s/it]
2025-09-07T09:45:57.0336651Z running benchmark:  23% 7/30 [00:11<00:36,  1.57s/it]
2025-09-07T09:45:58.6095166Z running benchmark:  27% 8/30 [00:12<00:34,  1.57s/it]
2025-09-07T09:46:00.1696843Z running benchmark:  30% 9/30 [00:14<00:33,  1.57s/it]
2025-09-07T09:46:01.7523812Z running benchmark:  33% 10/30 [00:15<00:31,  1.57s/it]
2025-09-07T09:46:03.3226412Z running benchmark:  37% 11/30 [00:17<00:29,  1.57s/it]
2025-09-07T09:46:04.9111341Z running benchmark:  40% 12/30 [00:18<00:28,  1.57s/it]
2025-09-07T09:46:06.5307916Z running benchmark:  43% 13/30 [00:20<00:26,  1.58s/it]
2025-09-07T09:46:08.1076753Z running benchmark:  47% 14/30 [00:22<00:25,  1.59s/it]
2025-09-07T09:46:09.7075874Z running benchmark:  50% 15/30 [00:23<00:23,  1.59s/it]
2025-09-07T09:46:11.3195838Z running benchmark:  53% 16/30 [00:25<00:22,  1.59s/it]
2025-09-07T09:46:12.8893390Z running benchmark:  57% 17/30 [00:26<00:20,  1.60s/it]
2025-09-07T09:46:14.4585663Z running benchmark:  60% 18/30 [00:28<00:19,  1.59s/it]
2025-09-07T09:46:16.0227093Z running benchmark:  63% 19/30 [00:30<00:17,  1.58s/it]
2025-09-07T09:46:17.6127335Z running benchmark:  67% 20/30 [00:31<00:15,  1.58s/it]
2025-09-07T09:46:19.1758073Z running benchmark:  70% 21/30 [00:33<00:14,  1.58s/it]
2025-09-07T09:46:20.7361263Z running benchmark:  73% 22/30 [00:34<00:12,  1.58s/it]
2025-09-07T09:46:22.3067191Z running benchmark:  77% 23/30 [00:36<00:10,  1.57s/it]
2025-09-07T09:46:23.8728936Z running benchmark:  80% 24/30 [00:37<00:09,  1.57s/it]
2025-09-07T09:46:25.4555548Z running benchmark:  83% 25/30 [00:39<00:07,  1.57s/it]
2025-09-07T09:46:27.0171189Z running benchmark:  87% 26/30 [00:41<00:06,  1.57s/it]
2025-09-07T09:46:28.6044499Z running benchmark:  90% 27/30 [00:42<00:04,  1.57s/it]
2025-09-07T09:46:30.2327050Z running benchmark:  93% 28/30 [00:44<00:03,  1.58s/it]
2025-09-07T09:46:31.7942278Z running benchmark:  97% 29/30 [00:45<00:01,  1.59s/it]
2025-09-07T09:46:31.7942589Z running benchmark: 100% 30/30 [00:47<00:00,  1.58s/it]
2025-09-07T09:46:31.7942852Z running benchmark: 100% 30/30 [00:47<00:00,  1.58s/it]
2025-09-07T09:46:32.1897351Z 1.550x
2025-09-07T09:46:32.1900474Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:46:35.7898508Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:46:35.7899419Z   import pynvml  # type: ignore[import]
2025-09-07T09:46:37.8067145Z 
2025-09-07T09:46:47.0486683Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:46:47.0487292Z loading model: 0it [00:09, ?it/s]
2025-09-07T09:46:47.1365246Z cpu  eval  hf_T5_large                        
2025-09-07T09:47:58.0362149Z 
2025-09-07T09:47:58.9375315Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:47:59.8169662Z running benchmark:   3% 1/30 [00:00<00:26,  1.11it/s]
2025-09-07T09:48:00.6866197Z running benchmark:   7% 2/30 [00:01<00:24,  1.13it/s]
2025-09-07T09:48:01.5545819Z running benchmark:  10% 3/30 [00:02<00:23,  1.14it/s]
2025-09-07T09:48:02.4283074Z running benchmark:  13% 4/30 [00:03<00:22,  1.14it/s]
2025-09-07T09:48:03.2901825Z running benchmark:  17% 5/30 [00:04<00:21,  1.14it/s]
2025-09-07T09:48:04.1937410Z running benchmark:  20% 6/30 [00:05<00:20,  1.15it/s]
2025-09-07T09:48:05.0577066Z running benchmark:  23% 7/30 [00:06<00:20,  1.13it/s]
2025-09-07T09:48:05.9314560Z running benchmark:  27% 8/30 [00:07<00:19,  1.14it/s]
2025-09-07T09:48:06.7953969Z running benchmark:  30% 9/30 [00:07<00:18,  1.14it/s]
2025-09-07T09:48:07.6664751Z running benchmark:  33% 10/30 [00:08<00:17,  1.15it/s]
2025-09-07T09:48:08.5365622Z running benchmark:  37% 11/30 [00:09<00:16,  1.15it/s]
2025-09-07T09:48:09.4097062Z running benchmark:  40% 12/30 [00:10<00:15,  1.15it/s]
2025-09-07T09:48:10.2770466Z running benchmark:  43% 13/30 [00:11<00:14,  1.15it/s]
2025-09-07T09:48:11.1971165Z running benchmark:  47% 14/30 [00:12<00:13,  1.15it/s]
2025-09-07T09:48:12.0599524Z running benchmark:  50% 15/30 [00:13<00:13,  1.13it/s]
2025-09-07T09:48:12.9315381Z running benchmark:  53% 16/30 [00:14<00:12,  1.14it/s]
2025-09-07T09:48:13.7964439Z running benchmark:  57% 17/30 [00:14<00:11,  1.14it/s]
2025-09-07T09:48:14.6692051Z running benchmark:  60% 18/30 [00:15<00:10,  1.15it/s]
2025-09-07T09:48:15.5305375Z running benchmark:  63% 19/30 [00:16<00:09,  1.15it/s]
2025-09-07T09:48:16.3984342Z running benchmark:  67% 20/30 [00:17<00:08,  1.15it/s]
2025-09-07T09:48:17.2608046Z running benchmark:  70% 21/30 [00:18<00:07,  1.15it/s]
2025-09-07T09:48:18.1296306Z running benchmark:  73% 22/30 [00:19<00:06,  1.15it/s]
2025-09-07T09:48:18.9979631Z running benchmark:  77% 23/30 [00:20<00:06,  1.15it/s]
2025-09-07T09:48:19.8815948Z running benchmark:  80% 24/30 [00:20<00:05,  1.15it/s]
2025-09-07T09:48:20.7620445Z running benchmark:  83% 25/30 [00:21<00:04,  1.15it/s]
2025-09-07T09:48:21.6310306Z running benchmark:  87% 26/30 [00:22<00:03,  1.14it/s]
2025-09-07T09:48:22.4892541Z running benchmark:  90% 27/30 [00:23<00:02,  1.15it/s]
2025-09-07T09:48:23.3581651Z running benchmark:  93% 28/30 [00:24<00:01,  1.15it/s]
2025-09-07T09:48:24.2309964Z running benchmark:  97% 29/30 [00:25<00:00,  1.15it/s]
2025-09-07T09:48:24.2310698Z running benchmark: 100% 30/30 [00:26<00:00,  1.15it/s]
2025-09-07T09:48:24.2310981Z running benchmark: 100% 30/30 [00:26<00:00,  1.15it/s]
2025-09-07T09:48:24.5528301Z 1.504x
2025-09-07T09:48:24.5532116Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:48:28.8633778Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:48:28.8634922Z   import pynvml  # type: ignore[import]
2025-09-07T09:48:30.8817441Z 
2025-09-07T09:48:35.5946705Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:48:35.5947399Z loading model: 0it [00:04, ?it/s]
2025-09-07T09:48:35.6266345Z cpu  eval  hf_distil_whisper                  
2025-09-07T09:49:04.1170463Z 
2025-09-07T09:49:04.7599854Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:49:05.3789756Z running benchmark:   3% 1/30 [00:00<00:18,  1.56it/s]
2025-09-07T09:49:05.9934733Z running benchmark:   7% 2/30 [00:01<00:17,  1.59it/s]
2025-09-07T09:49:06.6106079Z running benchmark:  10% 3/30 [00:01<00:16,  1.61it/s]
2025-09-07T09:49:07.2313278Z running benchmark:  13% 4/30 [00:02<00:16,  1.61it/s]
2025-09-07T09:49:07.8479636Z running benchmark:  17% 5/30 [00:03<00:15,  1.61it/s]
2025-09-07T09:49:08.4703038Z running benchmark:  20% 6/30 [00:03<00:14,  1.62it/s]
2025-09-07T09:49:09.0857078Z running benchmark:  23% 7/30 [00:04<00:14,  1.61it/s]
2025-09-07T09:49:09.7199285Z running benchmark:  27% 8/30 [00:04<00:13,  1.62it/s]
2025-09-07T09:49:10.3356868Z running benchmark:  30% 9/30 [00:05<00:13,  1.60it/s]
2025-09-07T09:49:10.9543016Z running benchmark:  33% 10/30 [00:06<00:12,  1.61it/s]
2025-09-07T09:49:11.5715886Z running benchmark:  37% 11/30 [00:06<00:11,  1.61it/s]
2025-09-07T09:49:12.1941639Z running benchmark:  40% 12/30 [00:07<00:11,  1.61it/s]
2025-09-07T09:49:12.8667602Z running benchmark:  43% 13/30 [00:08<00:10,  1.61it/s]
2025-09-07T09:49:13.4830862Z running benchmark:  47% 14/30 [00:08<00:10,  1.57it/s]
2025-09-07T09:49:14.0948152Z running benchmark:  50% 15/30 [00:09<00:09,  1.59it/s]
2025-09-07T09:49:14.7222612Z running benchmark:  53% 16/30 [00:09<00:08,  1.60it/s]
2025-09-07T09:49:15.3451541Z running benchmark:  57% 17/30 [00:10<00:08,  1.60it/s]
2025-09-07T09:49:15.9699663Z running benchmark:  60% 18/30 [00:11<00:07,  1.60it/s]
2025-09-07T09:49:16.5923664Z running benchmark:  63% 19/30 [00:11<00:06,  1.60it/s]
2025-09-07T09:49:17.2102387Z running benchmark:  67% 20/30 [00:12<00:06,  1.60it/s]
2025-09-07T09:49:17.8229698Z running benchmark:  70% 21/30 [00:13<00:05,  1.61it/s]
2025-09-07T09:49:18.4390236Z running benchmark:  73% 22/30 [00:13<00:04,  1.61it/s]
2025-09-07T09:49:19.0607933Z running benchmark:  77% 23/30 [00:14<00:04,  1.62it/s]
2025-09-07T09:49:19.6842455Z running benchmark:  80% 24/30 [00:14<00:03,  1.61it/s]
2025-09-07T09:49:20.3025529Z running benchmark:  83% 25/30 [00:15<00:03,  1.61it/s]
2025-09-07T09:49:20.9463838Z running benchmark:  87% 26/30 [00:16<00:02,  1.61it/s]
2025-09-07T09:49:21.5635993Z running benchmark:  90% 27/30 [00:16<00:01,  1.59it/s]
2025-09-07T09:49:22.1840712Z running benchmark:  93% 28/30 [00:17<00:01,  1.60it/s]
2025-09-07T09:49:22.8052096Z running benchmark:  97% 29/30 [00:18<00:00,  1.61it/s]
2025-09-07T09:49:22.8052490Z running benchmark: 100% 30/30 [00:18<00:00,  1.61it/s]
2025-09-07T09:49:22.8052790Z running benchmark: 100% 30/30 [00:18<00:00,  1.61it/s]
2025-09-07T09:49:22.9669637Z 1.206x
2025-09-07T09:49:22.9673608Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:49:25.6939974Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
2025-09-07T09:49:25.6941121Z   import pynvml  # type: ignore[import]
2025-09-07T09:49:27.7024394Z 
2025-09-07T09:49:27.8934738Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:49:27.8935019Z loading model: 0it [00:00, ?it/s]
2025-09-07T09:49:27.8935263Z cpu  eval  lennard_jones                      
2025-09-07T09:49:37.6079934Z 
2025-09-07T09:49:37.6822748Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2025-09-07T09:49:37.6823096Z running benchmark: 100% 30/30 [00:00<00:00, 406.06it/s]
2025-09-07T09:49:37.7262722Z 0.889x
2025-09-07T09:49:37.7266020Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu]
2025-09-07T09:49:38.7984048Z speedup             gmean=1.20x mean=1.230x
2025-09-07T09:49:38.7985134Z abs_latency         gmean=52.21x mean=154.229x
2025-09-07T09:49:38.7985390Z compilation_latency mean=25.618 seconds
2025-09-07T09:49:38.7985901Z compression_ratio   mean=0.738x
2025-09-07T09:49:38.7987874Z eager_peak_mem      gmean=239.60x mean=394.549x
2025-09-07T09:49:38.7990136Z dynamo_peak_mem     gmean=327.94x mean=549.397x
2025-09-07T09:49:38.7992293Z calls_captured      gmean=0.00x mean=0.000x
2025-09-07T09:49:38.7994416Z unique_graphs       gmean=0.00x mean=0.000x
2025-09-07T09:49:38.7996551Z graph_breaks        gmean=0.00x mean=0.000x
2025-09-07T09:49:38.7998683Z unique_graph_breaks gmean=0.00x mean=0.000x
2025-09-07T09:49:38.8000811Z autograd_captures   gmean=0.00x mean=0.000x
2025-09-07T09:49:38.8002992Z autograd_compiles   gmean=0.00x mean=0.000x
2025-09-07T09:49:38.8005227Z cudagraph_skips     gmean=0.00x mean=0.000x
2025-09-07T09:49:40.6173175Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *maxautotune-true* ]]
2025-09-07T09:49:40.6174447Z + [[ training-false-inference-true-default-true-dynamic-true-cppwrapper-true-aotinductor-true == *cudagraphs_low_precision-true* ]]
2025-09-07T09:49:40.6174936Z + sccache_epilogue
2025-09-07T09:49:40.6175153Z + echo '::group::Sccache Compilation Log'
2025-09-07T09:49:40.6175637Z ##[group]Sccache Compilation Log
2025-09-07T09:49:40.6175864Z + echo '=================== sccache compilation log ==================='
2025-09-07T09:49:40.6176119Z =================== sccache compilation log ===================
2025-09-07T09:49:40.6176515Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
2025-09-07T09:49:40.6436823Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
2025-09-07T09:49:40.6437239Z =========== If your build fails, please take a look at the log above for possible reasons ===========
2025-09-07T09:49:40.6437538Z + sccache --show-stats
2025-09-07T09:49:40.6507629Z Compile requests                   1980
2025-09-07T09:49:40.6507855Z Compile requests executed           127
2025-09-07T09:49:40.6508044Z Cache hits                           48
2025-09-07T09:49:40.6508247Z Cache hits (C/C++)                   48
2025-09-07T09:49:40.6508431Z Cache misses                         79
2025-09-07T09:49:40.6508605Z Cache misses (C/C++)                 79
2025-09-07T09:49:40.6508795Z Cache hits rate                   37.80 %
2025-09-07T09:49:40.6508993Z Cache hits rate (C/C++)           37.80 %
2025-09-07T09:49:40.6509337Z Cache timeouts                        0
2025-09-07T09:49:40.6509514Z Cache read errors                     0
2025-09-07T09:49:40.6509695Z Forced recaches                       0
2025-09-07T09:49:40.6509877Z Cache write errors                    0
2025-09-07T09:49:40.6510057Z Cache errors                          0
2025-09-07T09:49:40.6510233Z Compilations                         79
2025-09-07T09:49:40.6510421Z Compilation failures                  0
2025-09-07T09:49:40.6510611Z Non-cacheable compilations            0
2025-09-07T09:49:40.6510803Z Non-cacheable calls                 229
2025-09-07T09:49:40.6510980Z Non-compilation calls              1624
2025-09-07T09:49:40.6511170Z Unsupported compiler calls            0
2025-09-07T09:49:40.6511439Z Average cache write               0.045 s
2025-09-07T09:49:40.6511636Z Average compiler                  6.861 s
2025-09-07T09:49:40.6511820Z Average cache read hit            0.052 s
2025-09-07T09:49:40.6512016Z Failed distributed compilations       0
2025-09-07T09:49:40.6512149Z 
2025-09-07T09:49:40.6512212Z Non-cacheable reasons:
2025-09-07T09:49:40.6512378Z -E                                  197
2025-09-07T09:49:40.6512564Z unknown source language              32
2025-09-07T09:49:40.6512693Z 
2025-09-07T09:49:40.6512840Z Cache location                  s3, name: ossci-compiler-cache-circleci-v2, prefix: /
2025-09-07T09:49:40.6513118Z Version (client)                0.10.0
2025-09-07T09:49:40.6513318Z + sccache --stop-server
2025-09-07T09:49:40.6554555Z Stopping sccache server...
2025-09-07T09:49:40.6557871Z Compile requests                   1980
2025-09-07T09:49:40.6558070Z Compile requests executed           127
2025-09-07T09:49:40.6558263Z Cache hits                           48
2025-09-07T09:49:40.6558443Z Cache hits (C/C++)                   48
2025-09-07T09:49:40.6558615Z Cache misses                         79
2025-09-07T09:49:40.6558797Z Cache misses (C/C++)                 79
2025-09-07T09:49:40.6558982Z Cache hits rate                   37.80 %
2025-09-07T09:49:40.6559176Z Cache hits rate (C/C++)           37.80 %
2025-09-07T09:49:40.6559354Z Cache timeouts                        0
2025-09-07T09:49:40.6559540Z Cache read errors                     0
2025-09-07T09:49:40.6559716Z Forced recaches                       0
2025-09-07T09:49:40.6559896Z Cache write errors                    0
2025-09-07T09:49:40.6560069Z Cache errors                          0
2025-09-07T09:49:40.6560247Z Compilations                         79
2025-09-07T09:49:40.6560494Z Compilation failures                  0
2025-09-07T09:49:40.6560703Z Non-cacheable compilations            0
2025-09-07T09:49:40.6560895Z Non-cacheable calls                 229
2025-09-07T09:49:40.6561081Z Non-compilation calls              1624
2025-09-07T09:49:40.6561273Z Unsupported compiler calls            0
2025-09-07T09:49:40.6561465Z Average cache write               0.045 s
2025-09-07T09:49:40.6561659Z Average compiler                  6.861 s
2025-09-07T09:49:40.6561841Z Average cache read hit            0.052 s
2025-09-07T09:49:40.6562038Z Failed distributed compilations       0
2025-09-07T09:49:40.6562166Z 
2025-09-07T09:49:40.6562228Z Non-cacheable reasons:
2025-09-07T09:49:40.6562389Z -E                                  197
2025-09-07T09:49:40.6562564Z unknown source language              32
2025-09-07T09:49:40.6562689Z 
2025-09-07T09:49:40.6562827Z Cache location                  s3, name: ossci-compiler-cache-circleci-v2, prefix: /
2025-09-07T09:49:40.6563095Z Version (client)                0.10.0
2025-09-07T09:49:40.6563974Z + echo ::endgroup::
2025-09-07T09:49:40.6564281Z ##[endgroup]
2025-09-07T09:49:40.6564432Z + cleanup_workspace
2025-09-07T09:49:40.6564724Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2025-09-07T09:49:40.6565184Z sudo may print the following warning message that can be ignored. The chown command will still run.
2025-09-07T09:49:40.6565555Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2025-09-07T09:49:40.6565882Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2025-09-07T09:49:40.6566222Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2025-09-07T09:49:40.6566565Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2025-09-07T09:49:40.6566852Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2025-09-07T09:49:41.4802022Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main
2025-09-07T09:49:41.4802332Z with:
2025-09-07T09:49:41.4802503Z   benchmark-results-dir: test/test-reports
2025-09-07T09:49:41.4802714Z   dry-run: false
2025-09-07T09:49:41.4802875Z   schema-version: v3
2025-09-07T09:49:41.4803870Z   github-token: ***
2025-09-07T09:49:41.4804036Z env:
2025-09-07T09:49:41.4804189Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:41.4804508Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:41.4804826Z ##[endgroup]
2025-09-07T09:49:41.4818285Z ##[group]Run set -eux
2025-09-07T09:49:41.4818480Z set -eux
2025-09-07T09:49:41.4818621Z 
2025-09-07T09:49:41.4818772Z if [[ -n "" ]]; then
2025-09-07T09:49:41.4818955Z   source ""
2025-09-07T09:49:41.4819111Z fi
2025-09-07T09:49:41.4819337Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-09-07T09:49:41.4819631Z 
2025-09-07T09:49:41.4819774Z DEVICE_NAME=""
2025-09-07T09:49:41.4819943Z DEVICE_TYPE=""
2025-09-07T09:49:41.4820098Z 
2025-09-07T09:49:41.4820254Z if command -v nvidia-smi; then
2025-09-07T09:49:41.4820536Z   # NB: I'm using PyTorch here to get the device name, however, it needs to
2025-09-07T09:49:41.4820891Z   # install the correct version of PyTorch manually for now. Any PyTorch
2025-09-07T09:49:41.4821221Z   # version is fine, I just use 2.7.1 to satify PYPIDEP linter
2025-09-07T09:49:41.4821492Z   python3 -mpip install torch==2.7.1
2025-09-07T09:49:41.4821716Z elif command -v rocminfo; then
2025-09-07T09:49:41.4821981Z   # NB: Installing torch on ROCm runner with pip here causes CI to fail
2025-09-07T09:49:41.4822546Z   # with a memoryview is too large error only on MI300 runners. Is pip
2025-09-07T09:49:41.4822889Z   # version on ROCm runner there too old? As a workaround, let's use the
2025-09-07T09:49:41.4823188Z   # GPU device name coming from rocminfo instead
2025-09-07T09:49:41.4823414Z   DEVICE_NAME=rocm
2025-09-07T09:49:41.4823709Z   DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs)
2025-09-07T09:49:41.4824010Z fi
2025-09-07T09:49:41.4824147Z 
2025-09-07T09:49:41.4824322Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV
2025-09-07T09:49:41.4824575Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV
2025-09-07T09:49:41.4836000Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:41.4836244Z env:
2025-09-07T09:49:41.4836396Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:41.4836706Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:41.4837018Z ##[endgroup]
2025-09-07T09:49:41.4871500Z + [[ -n '' ]]
2025-09-07T09:49:41.4871842Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-09-07T09:49:41.6632924Z Defaulting to user installation because normal site-packages is not writeable
2025-09-07T09:49:42.4325850Z Collecting boto3==1.35.33
2025-09-07T09:49:42.4452287Z   Downloading boto3-1.35.33-py3-none-any.whl (139 kB)
2025-09-07T09:49:42.6601311Z Collecting psutil==7.0.0
2025-09-07T09:49:42.6631206Z   Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB)
2025-09-07T09:49:42.6915955Z Collecting pynvml==12.0.0
2025-09-07T09:49:42.6940260Z   Downloading pynvml-12.0.0-py3-none-any.whl (26 kB)
2025-09-07T09:49:42.7335348Z Collecting s3transfer<0.11.0,>=0.10.0
2025-09-07T09:49:42.7359916Z   Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB)
2025-09-07T09:49:42.7407456Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0)
2025-09-07T09:49:43.5231156Z Collecting botocore<1.36.0,>=1.35.33
2025-09-07T09:49:43.5263263Z   Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB)
2025-09-07T09:49:43.6645451Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0
2025-09-07T09:49:43.6669401Z   Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB)
2025-09-07T09:49:43.6734507Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1)
2025-09-07T09:49:43.6741787Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10)
2025-09-07T09:49:43.8125725Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0)
2025-09-07T09:49:43.9056856Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3
2025-09-07T09:49:44.3109477Z   Attempting uninstall: nvidia-ml-py
2025-09-07T09:49:44.3110643Z     Found existing installation: nvidia-ml-py 11.525.84
2025-09-07T09:49:44.3121411Z     Uninstalling nvidia-ml-py-11.525.84:
2025-09-07T09:49:44.3303867Z       Successfully uninstalled nvidia-ml-py-11.525.84
2025-09-07T09:49:44.3733026Z   Attempting uninstall: psutil
2025-09-07T09:49:44.3733650Z     Found existing installation: psutil 5.9.8
2025-09-07T09:49:44.3794816Z     Uninstalling psutil-5.9.8:
2025-09-07T09:49:44.3800139Z       Successfully uninstalled psutil-5.9.8
2025-09-07T09:49:44.5063401Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4
2025-09-07T09:49:44.5966572Z + DEVICE_NAME=
2025-09-07T09:49:44.5966803Z + DEVICE_TYPE=
2025-09-07T09:49:44.5966983Z + command -v nvidia-smi
2025-09-07T09:49:44.5967155Z + command -v rocminfo
2025-09-07T09:49:44.5967326Z + echo DEVICE_NAME=
2025-09-07T09:49:44.5968760Z + echo DEVICE_TYPE=
2025-09-07T09:49:44.5991014Z ##[group]Run set -eux
2025-09-07T09:49:44.5991221Z set -eux
2025-09-07T09:49:44.5991393Z 
2025-09-07T09:49:44.5991553Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2025-09-07T09:49:44.5991782Z   echo "Missing github-token input"
2025-09-07T09:49:44.5991985Z   exit 1
2025-09-07T09:49:44.5992133Z fi
2025-09-07T09:49:44.5999209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:44.5999463Z env:
2025-09-07T09:49:44.5999618Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:44.5999989Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:44.6000327Z   DEVICE_NAME: 
2025-09-07T09:49:44.6000477Z   DEVICE_TYPE: 
2025-09-07T09:49:44.6000849Z   GITHUB_TOKEN: ***
2025-09-07T09:49:44.6001015Z ##[endgroup]
2025-09-07T09:49:44.6032205Z + [[ -z *** ]]
2025-09-07T09:49:44.6064927Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2025-09-07T09:49:44.6065213Z with:
2025-09-07T09:49:44.6065473Z   github-token: ***
2025-09-07T09:49:44.6065633Z env:
2025-09-07T09:49:44.6065795Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:44.6066101Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:44.6066416Z   DEVICE_NAME: 
2025-09-07T09:49:44.6066568Z   DEVICE_TYPE: 
2025-09-07T09:49:44.6066708Z ##[endgroup]
2025-09-07T09:49:44.6077215Z ##[group]Run set -eux
2025-09-07T09:49:44.6077412Z set -eux
2025-09-07T09:49:44.6077560Z 
2025-09-07T09:49:44.6077856Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-09-07T09:49:44.6085329Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:44.6085666Z env:
2025-09-07T09:49:44.6085817Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:44.6086122Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:44.6086444Z   DEVICE_NAME: 
2025-09-07T09:49:44.6086593Z   DEVICE_TYPE: 
2025-09-07T09:49:44.6086849Z   GITHUB_TOKEN: ***
2025-09-07T09:49:44.6087165Z ##[endgroup]
2025-09-07T09:49:44.6114789Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 17525294857 i-0d9447236daff7b96
2025-09-07T09:49:45.0543347Z setting job-id=49775530527
2025-09-07T09:49:45.0543781Z setting job-name=inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T09:49:45.0637565Z ##[group]Run set -eux
2025-09-07T09:49:45.0637771Z set -eux
2025-09-07T09:49:45.0637927Z 
2025-09-07T09:49:45.0638078Z if [[ -n "" ]]; then
2025-09-07T09:49:45.0638276Z   source ""
2025-09-07T09:49:45.0638423Z fi
2025-09-07T09:49:45.0638562Z 
2025-09-07T09:49:45.0638811Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2025-09-07T09:49:45.0639131Z   --schema-version "${SCHEMA_VERSION}" \
2025-09-07T09:49:45.0639348Z   --repo "${REPO}" \
2025-09-07T09:49:45.0639552Z   --head-branch "${HEAD_BRANCH}" \
2025-09-07T09:49:45.0639764Z   --head-sha "${HEAD_SHA}" \
2025-09-07T09:49:45.0639977Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2025-09-07T09:49:45.0640200Z   --run-attempt "${RUN_ATTEMPT}" \
2025-09-07T09:49:45.0640405Z   --job-id "${JOB_ID}" \
2025-09-07T09:49:45.0640618Z   --job-name "${JOB_NAME}"
2025-09-07T09:49:45.0647727Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:45.0647972Z env:
2025-09-07T09:49:45.0648123Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:45.0648427Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:45.0648748Z   DEVICE_NAME: 
2025-09-07T09:49:45.0648891Z   DEVICE_TYPE: 
2025-09-07T09:49:45.0649039Z   SCHEMA_VERSION: v3
2025-09-07T09:49:45.0649211Z   REPO: pytorch/pytorch
2025-09-07T09:49:45.0649385Z   HEAD_BRANCH: refs/heads/main
2025-09-07T09:49:45.0649593Z   HEAD_SHA: 93fb23d6fae7c4e82c4239a1033e522088742634
2025-09-07T09:49:45.0649810Z   WORKFLOW_RUN_ID: 17525294857
2025-09-07T09:49:45.0649980Z   RUN_ATTEMPT: 1
2025-09-07T09:49:45.0650129Z   JOB_ID: 49775530527
2025-09-07T09:49:45.0650441Z   JOB_NAME: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T09:49:45.0650786Z ##[endgroup]
2025-09-07T09:49:45.0682151Z + [[ -n '' ]]
2025-09-07T09:49:45.0683391Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 93fb23d6fae7c4e82c4239a1033e522088742634 --workflow-id 17525294857 --run-attempt 1 --job-id 49775530527 --job-name 'inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)'
2025-09-07T09:49:45.0955513Z ##[group]Run set -eux
2025-09-07T09:49:45.0955922Z set -eux
2025-09-07T09:49:45.0956177Z 
2025-09-07T09:49:45.0956512Z if [[ -n "" ]]; then
2025-09-07T09:49:45.0956865Z   source ""
2025-09-07T09:49:45.0957159Z fi
2025-09-07T09:49:45.0957438Z 
2025-09-07T09:49:45.0957842Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py"
2025-09-07T09:49:45.0965334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:45.0965721Z env:
2025-09-07T09:49:45.0966047Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:45.0966479Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:45.0967130Z   DEVICE_NAME: 
2025-09-07T09:49:45.0967418Z   DEVICE_TYPE: 
2025-09-07T09:49:45.0967702Z ##[endgroup]
2025-09-07T09:49:45.0997097Z + [[ -n '' ]]
2025-09-07T09:49:45.0997851Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py
2025-09-07T09:49:45.1324031Z INFO:root:Fail to import torch to get the device name
2025-09-07T09:49:45.1442951Z ##[group]Run set -eux
2025-09-07T09:49:45.1443234Z set -eux
2025-09-07T09:49:45.1443571Z 
2025-09-07T09:49:45.1443860Z # TODO (huydhn): Implement this part
2025-09-07T09:49:45.1444208Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2025-09-07T09:49:45.1451759Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:45.1452103Z env:
2025-09-07T09:49:45.1452563Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:45.1453022Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:45.1453494Z   DEVICE_NAME: 
2025-09-07T09:49:45.1453807Z   DEVICE_TYPE: 
2025-09-07T09:49:45.1454096Z ##[endgroup]
2025-09-07T09:49:45.1483311Z + echo 'dependencies={}'
2025-09-07T09:49:45.1509615Z ##[group]Run set -eux
2025-09-07T09:49:45.1509992Z set -eux
2025-09-07T09:49:45.1510292Z 
2025-09-07T09:49:45.1510643Z if [[ -n "" ]]; then
2025-09-07T09:49:45.1510958Z   source ""
2025-09-07T09:49:45.1511224Z fi
2025-09-07T09:49:45.1511552Z 
2025-09-07T09:49:45.1511864Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2025-09-07T09:49:45.1512262Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2025-09-07T09:49:45.1512726Z   # We don't want the job to fail if the directory doesn't exist
2025-09-07T09:49:45.1513231Z   exit 0
2025-09-07T09:49:45.1513508Z fi
2025-09-07T09:49:45.1513851Z 
2025-09-07T09:49:45.1514159Z if [[ "${DRY_RUN}" == "true" ]]; then
2025-09-07T09:49:45.1514544Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-09-07T09:49:45.1515097Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-09-07T09:49:45.1515522Z     --metadata "${BENCHMARK_METADATA}" \
2025-09-07T09:49:45.1515905Z     --runners "${RUNNER_INFO}" \
2025-09-07T09:49:45.1516242Z     --dependencies "${DEPENDENCIES}" \
2025-09-07T09:49:45.1516604Z     --dry-run
2025-09-07T09:49:45.1516835Z else
2025-09-07T09:49:45.1517066Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-09-07T09:49:45.1517404Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-09-07T09:49:45.1517665Z     --metadata "${BENCHMARK_METADATA}" \
2025-09-07T09:49:45.1517897Z     --runners "${RUNNER_INFO}" \
2025-09-07T09:49:45.1518112Z     --dependencies "${DEPENDENCIES}"
2025-09-07T09:49:45.1518314Z fi
2025-09-07T09:49:45.1525255Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:45.1525494Z env:
2025-09-07T09:49:45.1525639Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:45.1525944Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:45.1526255Z   DEVICE_NAME: 
2025-09-07T09:49:45.1526413Z   DEVICE_TYPE: 
2025-09-07T09:49:45.1526578Z   BENCHMARK_RESULTS_DIR: test/test-reports
2025-09-07T09:49:45.1526771Z   DRY_RUN: false
2025-09-07T09:49:45.1527581Z   BENCHMARK_METADATA: {"timestamp": 1757238585, "schema_version": "v3", "name": "inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "93fb23d6fae7c4e82c4239a1033e522088742634", "workflow_id": 17525294857, "run_attempt": 1, "job_id": 49775530527}
2025-09-07T09:49:45.1528622Z   RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 96, "avail_mem_in_gb": 369, "extra_info": {"hostname": "ip-10-0-57-198.ec2.internal"}, "name": "", "type": ""}]
2025-09-07T09:49:45.1529113Z   DEPENDENCIES: {}
2025-09-07T09:49:45.1529275Z ##[endgroup]
2025-09-07T09:49:45.1555931Z + [[ -n '' ]]
2025-09-07T09:49:45.1556118Z + [[ ! -d test/test-reports ]]
2025-09-07T09:49:45.1556302Z + [[ false == \t\r\u\e ]]
2025-09-07T09:49:45.1558176Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1757238585, "schema_version": "v3", "name": "inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "93fb23d6fae7c4e82c4239a1033e522088742634", "workflow_id": 17525294857, "run_attempt": 1, "job_id": 49775530527}' --runners '[{"cpu_info": "x86_64", "cpu_count": 96, "avail_mem_in_gb": 369, "extra_info": {"hostname": "ip-10-0-57-198.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}'
2025-09-07T09:49:45.2668415Z INFO:root:Upload test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json
2025-09-07T09:49:45.2950254Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role
2025-09-07T09:49:45.4490138Z INFO:root:Upload test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json
2025-09-07T09:49:45.5256957Z INFO:root:Upload test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json
2025-09-07T09:49:45.6117090Z INFO:root:Upload test/test-reports/inductor_export_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_export_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json
2025-09-07T09:49:45.7111758Z INFO:root:Upload test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json
2025-09-07T09:49:45.7896491Z INFO:root:Upload test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance.json
2025-09-07T09:49:45.9113607Z INFO:root:Upload test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json
2025-09-07T09:49:46.0193871Z INFO:root:Upload test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance.json
2025-09-07T09:49:46.1210589Z INFO:root:Upload test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json
2025-09-07T09:49:46.2179284Z INFO:root:Upload test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance.json
2025-09-07T09:49:46.3531781Z INFO:root:Upload test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json
2025-09-07T09:49:46.4908269Z INFO:root:Upload test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance.json
2025-09-07T09:49:46.6270742Z INFO:root:Upload test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json to s3://ossci-benchmarks/v3/pytorch/pytorch/17525294857/49775530527/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json
2025-09-07T09:49:46.7607100Z ##[group]Run cat test/**/*_toprint.log || true
2025-09-07T09:49:46.7607376Z cat test/**/*_toprint.log || true
2025-09-07T09:49:46.7614380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:46.7614617Z env:
2025-09-07T09:49:46.7614766Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:46.7615069Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:46.7615378Z   DEVICE_NAME: 
2025-09-07T09:49:46.7615526Z   DEVICE_TYPE: 
2025-09-07T09:49:46.7615671Z ##[endgroup]
2025-09-07T09:49:46.7699287Z cat: 'test/**/*_toprint.log': No such file or directory
2025-09-07T09:49:46.7723842Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2025-09-07T09:49:46.7724116Z kill "$MONITOR_SCRIPT_PID"
2025-09-07T09:49:46.7731160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:46.7731396Z env:
2025-09-07T09:49:46.7731545Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:46.7731861Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:46.7732171Z   DEVICE_NAME: 
2025-09-07T09:49:46.7732323Z   DEVICE_TYPE: 
2025-09-07T09:49:46.7732474Z   MONITOR_SCRIPT_PID: 57444
2025-09-07T09:49:46.7732667Z ##[endgroup]
2025-09-07T09:49:46.7834628Z Prepare all required actions
2025-09-07T09:49:46.7834956Z Getting action download info
2025-09-07T09:49:46.9287482Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-09-07T09:49:47.1899403Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-09-07T09:49:47.5909181Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-09-07T09:49:47.5909428Z with:
2025-09-07T09:49:47.5909705Z   file-suffix: test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T09:49:47.5910030Z   s3-bucket: gha-artifacts
2025-09-07T09:49:47.5910195Z env:
2025-09-07T09:49:47.5910347Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:47.5910644Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:47.5910963Z   DEVICE_NAME: 
2025-09-07T09:49:47.5911112Z   DEVICE_TYPE: 
2025-09-07T09:49:47.5911252Z ##[endgroup]
2025-09-07T09:49:47.5927757Z ##[group]Run # Remove any previous test jsons if they exist
2025-09-07T09:49:47.5928047Z # Remove any previous test jsons if they exist
2025-09-07T09:49:47.5928278Z rm -f test-jsons-*.zip
2025-09-07T09:49:47.5928548Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json'
2025-09-07T09:49:47.5935466Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:47.5935704Z env:
2025-09-07T09:49:47.5935846Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:47.5936232Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:47.5936544Z   DEVICE_NAME: 
2025-09-07T09:49:47.5936695Z   DEVICE_TYPE: 
2025-09-07T09:49:47.5937067Z   FILE_SUFFIX: test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T09:49:47.5937363Z ##[endgroup]
2025-09-07T09:49:47.6075397Z   adding: test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json (deflated 99%)
2025-09-07T09:49:47.6102725Z   adding: test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json (deflated 99%)
2025-09-07T09:49:47.6130467Z   adding: test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json (deflated 99%)
2025-09-07T09:49:47.6157928Z   adding: test/test-reports/inductor_export_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json (deflated 99%)
2025-09-07T09:49:47.6178494Z   adding: test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.json (deflated 99%)
2025-09-07T09:49:47.6220535Z   adding: test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance.json (deflated 99%)
2025-09-07T09:49:47.6315276Z   adding: test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json (deflated 99%)
2025-09-07T09:49:47.6357052Z   adding: test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance.json (deflated 99%)
2025-09-07T09:49:47.6452444Z   adding: test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json (deflated 99%)
2025-09-07T09:49:47.6493934Z   adding: test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance.json (deflated 99%)
2025-09-07T09:49:47.6588783Z   adding: test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json (deflated 99%)
2025-09-07T09:49:47.6619292Z   adding: test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance.json (deflated 99%)
2025-09-07T09:49:47.6664403Z   adding: test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.json (deflated 99%)
2025-09-07T09:49:47.6689566Z ##[group]Run # Remove any previous test reports if they exist
2025-09-07T09:49:47.6690108Z # Remove any previous test reports if they exist
2025-09-07T09:49:47.6690367Z rm -f test-reports-*.zip
2025-09-07T09:49:47.6690658Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv'
2025-09-07T09:49:47.6697918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:47.6698160Z env:
2025-09-07T09:49:47.6698309Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:47.6698629Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:47.6698953Z   DEVICE_NAME: 
2025-09-07T09:49:47.6699099Z   DEVICE_TYPE: 
2025-09-07T09:49:47.6699368Z   FILE_SUFFIX: test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T09:49:47.6699681Z ##[endgroup]
2025-09-07T09:49:47.6763871Z   adding: test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv (deflated 59%)
2025-09-07T09:49:47.6764472Z   adding: test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv (deflated 58%)
2025-09-07T09:49:47.6765029Z   adding: test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv (deflated 59%)
2025-09-07T09:49:47.6765567Z   adding: test/test-reports/inductor_export_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv (deflated 63%)
2025-09-07T09:49:47.6766107Z   adding: test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_accuracy.csv (deflated 71%)
2025-09-07T09:49:47.6766682Z   adding: test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv (deflated 50%)
2025-09-07T09:49:47.6768911Z   adding: test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.csv (deflated 51%)
2025-09-07T09:49:47.6769647Z   adding: test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv (deflated 50%)
2025-09-07T09:49:47.6772149Z   adding: test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.csv (deflated 51%)
2025-09-07T09:49:47.6772763Z   adding: test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv (deflated 50%)
2025-09-07T09:49:47.6820415Z   adding: test/test-reports/inductor_cpp_wrapper_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.csv (deflated 51%)
2025-09-07T09:49:47.6821271Z   adding: test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance.csv (deflated 52%)
2025-09-07T09:49:47.6821896Z   adding: test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cpu_x86_zen_performance_compilation_metrics.csv (deflated 49%)
2025-09-07T09:49:47.6836994Z ##[group]Run # Remove any previous usage logs if they exist
2025-09-07T09:49:47.6837288Z # Remove any previous usage logs if they exist
2025-09-07T09:49:47.6837521Z rm -f logs-*.zip
2025-09-07T09:49:47.6837752Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true
2025-09-07T09:49:47.6838070Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true
2025-09-07T09:49:47.6844208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:47.6844453Z env:
2025-09-07T09:49:47.6844605Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:47.6844912Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:47.6845226Z   DEVICE_NAME: 
2025-09-07T09:49:47.6845381Z   DEVICE_TYPE: 
2025-09-07T09:49:47.6845648Z   FILE_SUFFIX: test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T09:49:47.6845954Z ##[endgroup]
2025-09-07T09:49:47.6942505Z   adding: usage_log.txt (deflated 96%)
2025-09-07T09:49:47.6956599Z 
2025-09-07T09:49:47.6956879Z zip error: Nothing to do! (logs-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip)
2025-09-07T09:49:47.6977804Z ##[group]Run # Remove any previous debugging artifacts if they exist
2025-09-07T09:49:47.6978187Z # Remove any previous debugging artifacts if they exist
2025-09-07T09:49:47.6978455Z rm -f debug-*.zip
2025-09-07T09:49:47.6978643Z if [ -d 'test/debug' ]; then
2025-09-07T09:49:47.6978870Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2025-09-07T09:49:47.6979083Z fi
2025-09-07T09:49:47.6986049Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:47.6986300Z env:
2025-09-07T09:49:47.6986447Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:47.6986759Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:47.6987088Z   DEVICE_NAME: 
2025-09-07T09:49:47.6987249Z   DEVICE_TYPE: 
2025-09-07T09:49:47.6987513Z   FILE_SUFFIX: test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527
2025-09-07T09:49:47.6987814Z ##[endgroup]
2025-09-07T09:49:47.7059082Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-09-07T09:49:47.7059315Z with:
2025-09-07T09:49:47.7059476Z   s3-bucket: gha-artifacts
2025-09-07T09:49:47.7059699Z   s3-prefix: pytorch/pytorch/17525294857/1/artifact

2025-09-07T09:49:47.7059985Z   retention-days: 14
2025-09-07T09:49:47.7060149Z   if-no-files-found: warn
2025-09-07T09:49:47.7060344Z   path: test-jsons-*.zip
2025-09-07T09:49:47.7060523Z   name: artifact
2025-09-07T09:49:47.7060683Z   region: us-east-1
2025-09-07T09:49:47.7060832Z env:
2025-09-07T09:49:47.7061007Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:47.7061318Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:47.7061635Z   DEVICE_NAME: 
2025-09-07T09:49:47.7061781Z   DEVICE_TYPE: 
2025-09-07T09:49:47.7061933Z ##[endgroup]
2025-09-07T09:49:47.9847141Z NOTE: s3-prefix specified, ignoring name parameter
2025-09-07T09:49:47.9847474Z With the provided path, there will be 1 file uploaded
2025-09-07T09:49:47.9847768Z Uploading to s3 prefix: pytorch/pytorch/17525294857/1/artifact
2025-09-07T09:49:47.9905372Z Starting upload of test-jsons-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:48.1117822Z Finished upload of test-jsons-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:48.1329809Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-09-07T09:49:48.1330040Z with:
2025-09-07T09:49:48.1330195Z   s3-bucket: gha-artifacts
2025-09-07T09:49:48.1330536Z   s3-prefix: pytorch/pytorch/17525294857/1/artifact

2025-09-07T09:49:48.1330764Z   retention-days: 14
2025-09-07T09:49:48.1330923Z   if-no-files-found: error
2025-09-07T09:49:48.1331098Z   path: test-reports-*.zip
2025-09-07T09:49:48.1331262Z   name: artifact
2025-09-07T09:49:48.1331411Z   region: us-east-1
2025-09-07T09:49:48.1331563Z env:
2025-09-07T09:49:48.1331706Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:48.1332009Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:48.1332334Z   DEVICE_NAME: 
2025-09-07T09:49:48.1332487Z   DEVICE_TYPE: 
2025-09-07T09:49:48.1332631Z ##[endgroup]
2025-09-07T09:49:48.4007370Z NOTE: s3-prefix specified, ignoring name parameter
2025-09-07T09:49:48.4007696Z With the provided path, there will be 1 file uploaded
2025-09-07T09:49:48.4007990Z Uploading to s3 prefix: pytorch/pytorch/17525294857/1/artifact
2025-09-07T09:49:48.4064788Z Starting upload of test-reports-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:48.5144202Z Finished upload of test-reports-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:48.5355807Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-09-07T09:49:48.5356030Z with:
2025-09-07T09:49:48.5356198Z   s3-bucket: gha-artifacts
2025-09-07T09:49:48.5356420Z   s3-prefix: pytorch/pytorch/17525294857/1/artifact

2025-09-07T09:49:48.5356643Z   retention-days: 14
2025-09-07T09:49:48.5356802Z   if-no-files-found: ignore
2025-09-07T09:49:48.5356978Z   path: logs-*.zip
2025-09-07T09:49:48.5357334Z   name: artifact
2025-09-07T09:49:48.5357489Z   region: us-east-1
2025-09-07T09:49:48.5357632Z env:
2025-09-07T09:49:48.5357774Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:48.5358083Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:48.5358406Z   DEVICE_NAME: 
2025-09-07T09:49:48.5358553Z   DEVICE_TYPE: 
2025-09-07T09:49:48.5358702Z ##[endgroup]
2025-09-07T09:49:48.8055113Z NOTE: s3-prefix specified, ignoring name parameter
2025-09-07T09:49:48.8055429Z With the provided path, there will be 1 file uploaded
2025-09-07T09:49:48.8055713Z Uploading to s3 prefix: pytorch/pytorch/17525294857/1/artifact
2025-09-07T09:49:48.8112663Z Starting upload of logs-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:48.9207493Z Finished upload of logs-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:48.9420368Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-09-07T09:49:48.9420612Z with:
2025-09-07T09:49:48.9420767Z   s3-bucket: gha-artifacts
2025-09-07T09:49:48.9420981Z   s3-prefix: pytorch/pytorch/17525294857/1/artifact

2025-09-07T09:49:48.9421203Z   retention-days: 14
2025-09-07T09:49:48.9421368Z   if-no-files-found: ignore
2025-09-07T09:49:48.9421546Z   path: debug-*.zip
2025-09-07T09:49:48.9421688Z   name: artifact
2025-09-07T09:49:48.9421838Z   region: us-east-1
2025-09-07T09:49:48.9421995Z env:
2025-09-07T09:49:48.9422134Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:48.9422430Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:48.9422752Z   DEVICE_NAME: 
2025-09-07T09:49:48.9422906Z   DEVICE_TYPE: 
2025-09-07T09:49:48.9423171Z ##[endgroup]
2025-09-07T09:49:49.2060229Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2025-09-07T09:49:49.2326056Z ##[group]Run # shellcheck disable=SC2156
2025-09-07T09:49:49.2326330Z # shellcheck disable=SC2156
2025-09-07T09:49:49.2326700Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-09-07T09:49:49.2333961Z shell: /usr/bin/bash -e {0}
2025-09-07T09:49:49.2334146Z env:
2025-09-07T09:49:49.2334287Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:49.2334597Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:49.2334914Z   DEVICE_NAME: 
2025-09-07T09:49:49.2335166Z   DEVICE_TYPE: 
2025-09-07T09:49:49.2335308Z ##[endgroup]
2025-09-07T09:49:49.5452068Z Prepare all required actions
2025-09-07T09:49:49.5452339Z Getting action download info
2025-09-07T09:49:49.6849870Z ##[group]Run ./.github/actions/upload-utilization-stats
2025-09-07T09:49:49.6850115Z with:
2025-09-07T09:49:49.6850263Z   job_id: 49775530527
2025-09-07T09:49:49.6850578Z   job_name: inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T09:49:49.6850948Z   workflow_name: inductor-perf-nightly-x86-zen
2025-09-07T09:49:49.6851167Z   workflow_run_id: 17525294857
2025-09-07T09:49:49.6851339Z   workflow_attempt: 1
2025-09-07T09:49:49.6851492Z env:
2025-09-07T09:49:49.6851635Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:49.6851933Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:49.6852247Z   DEVICE_NAME: 
2025-09-07T09:49:49.6852397Z   DEVICE_TYPE: 
2025-09-07T09:49:49.6852546Z ##[endgroup]
2025-09-07T09:49:49.6864369Z ##[group]Run echo "workflow_id: 17525294857"
2025-09-07T09:49:49.6864630Z echo "workflow_id: 17525294857"
2025-09-07T09:49:49.6864846Z echo "workflow_attempt: 1"
2025-09-07T09:49:49.6865091Z echo "workflow_Name: inductor-perf-nightly-x86-zen"
2025-09-07T09:49:49.6865333Z echo "job_id: 49775530527"
2025-09-07T09:49:49.6865694Z echo "job_name:  inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)"
2025-09-07T09:49:49.6866294Z echo "artifact_prefix: "
2025-09-07T09:49:49.6866497Z python3 --version
2025-09-07T09:49:49.6873675Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:49.6873912Z env:
2025-09-07T09:49:49.6874060Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:49.6874361Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:49.6874680Z   DEVICE_NAME: 
2025-09-07T09:49:49.6874831Z   DEVICE_TYPE: 
2025-09-07T09:49:49.6874971Z ##[endgroup]
2025-09-07T09:49:49.6900821Z workflow_id: 17525294857
2025-09-07T09:49:49.6901001Z workflow_attempt: 1
2025-09-07T09:49:49.6901205Z workflow_Name: inductor-perf-nightly-x86-zen
2025-09-07T09:49:49.6901424Z job_id: 49775530527
2025-09-07T09:49:49.6901741Z job_name:  inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)
2025-09-07T09:49:49.6902082Z artifact_prefix: 
2025-09-07T09:49:49.6916622Z Python 3.9.23
2025-09-07T09:49:49.6947864Z ##[group]Run nick-fields/retry@v3.0.0
2025-09-07T09:49:49.6948082Z with:
2025-09-07T09:49:49.6948217Z   shell: bash
2025-09-07T09:49:49.6948368Z   timeout_minutes: 5
2025-09-07T09:49:49.6948530Z   max_attempts: 5
2025-09-07T09:49:49.6948698Z   retry_wait_seconds: 30
2025-09-07T09:49:49.6949042Z   command: set -eu
python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7

2025-09-07T09:49:49.6949408Z   polling_interval_seconds: 1
2025-09-07T09:49:49.6949612Z   warning_on_retry: true
2025-09-07T09:49:49.6949805Z   continue_on_error: false
2025-09-07T09:49:49.6949973Z env:
2025-09-07T09:49:49.6950109Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:49.6950408Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:49.6950856Z   DEVICE_NAME: 
2025-09-07T09:49:49.6951007Z   DEVICE_TYPE: 
2025-09-07T09:49:49.6951143Z ##[endgroup]
2025-09-07T09:49:49.9545213Z Defaulting to user installation because normal site-packages is not writeable
2025-09-07T09:49:50.0154190Z Collecting python-dateutil==2.8.2
2025-09-07T09:49:50.0454453Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2025-09-07T09:49:50.7129406Z Collecting boto3==1.35.42
2025-09-07T09:49:50.7220061Z   Downloading boto3-1.35.42-py3-none-any.whl (139 kB)
2025-09-07T09:49:51.0747637Z Collecting pandas==2.1.3
2025-09-07T09:49:51.0844527Z   Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)
2025-09-07T09:49:51.1927257Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7)
2025-09-07T09:49:51.1937947Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0)
2025-09-07T09:49:51.1968704Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99)
2025-09-07T09:49:51.1972336Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0)
2025-09-07T09:49:51.1974837Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4)
2025-09-07T09:49:51.2610699Z Collecting tzdata>=2022.1
2025-09-07T09:49:51.2698121Z   Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB)
2025-09-07T09:49:51.8284387Z Collecting numpy<2,>=1.22.4
2025-09-07T09:49:51.8376523Z   Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
2025-09-07T09:49:51.9780934Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1)
2025-09-07T09:49:51.9801818Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1)
2025-09-07T09:49:51.9804830Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0)
2025-09-07T09:49:51.9847245Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10)
2025-09-07T09:49:51.9923636Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0)
2025-09-07T09:49:51.9993504Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0)
2025-09-07T09:49:51.9995850Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.15.0)
2025-09-07T09:49:52.1362304Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3
2025-09-07T09:49:55.8870493Z   Attempting uninstall: boto3
2025-09-07T09:49:55.8870822Z     Found existing installation: boto3 1.35.33
2025-09-07T09:49:55.8961321Z     Uninstalling boto3-1.35.33:
2025-09-07T09:49:55.8973446Z       Successfully uninstalled boto3-1.35.33
2025-09-07T09:49:55.9419179Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2
2025-09-07T09:49:56.7622889Z Command completed after 1 attempt(s).
2025-09-07T09:49:56.7679873Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \
2025-09-07T09:49:56.7680342Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \
2025-09-07T09:49:56.7680651Z   --workflow-run-id "17525294857" \
2025-09-07T09:49:56.7680919Z   --workflow-name "inductor-perf-nightly-x86-zen" \
2025-09-07T09:49:56.7681289Z   --workflow-run-attempt "1" \
2025-09-07T09:49:56.7681497Z   --job-id "49775530527" \
2025-09-07T09:49:56.7681861Z   --job-name "inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)" \
2025-09-07T09:49:56.7682219Z   --local-path "" \
2025-09-07T09:49:56.7682413Z   --artifact-prefix ""
2025-09-07T09:49:56.7690454Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:56.7690692Z env:
2025-09-07T09:49:56.7690837Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:56.7691139Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:56.7691449Z   DEVICE_NAME: 
2025-09-07T09:49:56.7691691Z   DEVICE_TYPE: 
2025-09-07T09:49:56.7691834Z ##[endgroup]
2025-09-07T09:49:57.6565015Z repo: pytorch/pytorch
2025-09-07T09:49:57.6565355Z Search for test log in s3 bucket: ossci-utilization
2025-09-07T09:49:57.6565747Z Downloading logs-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:57.6566285Z extracting usage_log.txt from zip file logs-test-inductor_torchbench_perf_cpu_x86_zen-2-4-linux.24xlarge.amd_49775530527.zip
2025-09-07T09:49:57.6566683Z Converted Log Model: UtilizationMetadata:
2025-09-07T09:49:57.6567602Z  UtilizationMetadata(level='metadata', workflow_id='17525294857', job_id='49775530527', workflow_name='inductor-perf-nightly-x86-zen', job_name='inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)', usage_collect_interval=4.0, data_model_version=1.5, start_at=1757231017, gpu_count=0, cpu_count=96, gpu_type=None, error=None)
2025-09-07T09:49:57.6568569Z [Db Segments] detected pytest cmd: 14, generated segments: 14
2025-09-07T09:49:57.6568832Z [db model] Peek db timeseries 
2025-09-07T09:49:57.6569009Z :{
2025-09-07T09:49:57.6569149Z     "created_at": 1757238597,
2025-09-07T09:49:57.6569328Z     "type": "utilization",
2025-09-07T09:49:57.6569486Z     "tags": [
2025-09-07T09:49:57.6569627Z         "record"
2025-09-07T09:49:57.6569766Z     ],
2025-09-07T09:49:57.6569906Z     "time_stamp": 1757231017,
2025-09-07T09:49:57.6570078Z     "repo": "pytorch/pytorch",
2025-09-07T09:49:57.6570265Z     "workflow_id": 17525294857,
2025-09-07T09:49:57.6570443Z     "run_attempt": 1,
2025-09-07T09:49:57.6570604Z     "job_id": 49775530527,
2025-09-07T09:49:57.6570808Z     "workflow_name": "inductor-perf-nightly-x86-zen",
2025-09-07T09:49:57.6571188Z     "job_name": "inductor-test-nightly / test (inductor_torchbench_perf_cpu_x86_zen, 2, 4, linux.24xlarge.amd)",
2025-09-07T09:49:57.6571527Z     "json_data": "{}"
2025-09-07T09:49:57.6571676Z }
2025-09-07T09:49:57.6571984Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/17525294857/1/49775530527/metadata
2025-09-07T09:49:57.6572548Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/17525294857/1/49775530527/metadata 
2025-09-07T09:49:57.6573113Z Writing 498 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/17525294857/1/49775530527/time_series
2025-09-07T09:49:57.6573678Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/17525294857/1/49775530527/time_series 
2025-09-07T09:49:57.7445370Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2025-09-07T09:49:57.7445689Z with:
2025-09-07T09:49:57.7446108Z env:
2025-09-07T09:49:57.7446254Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:57.7446568Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:57.7446895Z   DEVICE_NAME: 
2025-09-07T09:49:57.7447049Z   DEVICE_TYPE: 
2025-09-07T09:49:57.7447186Z ##[endgroup]
2025-09-07T09:49:57.7468758Z ##[group]Run set -eou pipefail
2025-09-07T09:49:57.7469044Z set -eou pipefail
2025-09-07T09:49:57.7469226Z 
2025-09-07T09:49:57.7469462Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2025-09-07T09:49:57.7469863Z for _ in $(seq 1440); do
2025-09-07T09:49:57.7470078Z     # Break if no ssh session exists anymore
2025-09-07T09:49:57.7470303Z     if [ "$(who)" = "" ]; then
2025-09-07T09:49:57.7470491Z       break
2025-09-07T09:49:57.7470674Z     fi
2025-09-07T09:49:57.7470828Z     echo "."
2025-09-07T09:49:57.7470983Z     sleep 5
2025-09-07T09:49:57.7471137Z done
2025-09-07T09:49:57.7478409Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:57.7478647Z env:
2025-09-07T09:49:57.7478786Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:57.7479093Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:57.7479517Z   DEVICE_NAME: 
2025-09-07T09:49:57.7479665Z   DEVICE_TYPE: 
2025-09-07T09:49:57.7479804Z ##[endgroup]
2025-09-07T09:49:57.7506260Z Holding runner for 2 hours until all ssh sessions have logged out
2025-09-07T09:49:57.7647222Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-09-07T09:49:57.7647619Z # ignore expansion of "docker ps -q" since it could be empty
2025-09-07T09:49:57.7647902Z # shellcheck disable=SC2046
2025-09-07T09:49:57.7648147Z docker stop $(docker ps -q) || true
2025-09-07T09:49:57.7648364Z # Prune all of the docker images
2025-09-07T09:49:57.7648576Z docker system prune -af
2025-09-07T09:49:57.7656272Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:49:57.7656521Z env:
2025-09-07T09:49:57.7656662Z   GIT_DEFAULT_BRANCH: main
2025-09-07T09:49:57.7656984Z   DOCKER_CONTAINER_ID: de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:49:57.7657324Z   DEVICE_NAME: 
2025-09-07T09:49:57.7657549Z   DEVICE_TYPE: 
2025-09-07T09:49:57.7657696Z ##[endgroup]
2025-09-07T09:50:09.0281794Z de0caba1838c
2025-09-07T09:50:10.5891194Z Deleted Containers:
2025-09-07T09:50:10.5891532Z de0caba1838c50b38cd32bd07459a7acfcd20c5f8c2aeab4a1af2a5c7c9a3fac
2025-09-07T09:50:10.5891774Z 
2025-09-07T09:50:26.4665019Z Deleted Images:
2025-09-07T09:50:26.4665678Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3-gcc11-inductor-benchmarks-ae53c6842aa4c2407d0ad976491ca941c2635c77
2025-09-07T09:50:26.4666493Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:383efb45082f20b8c808cb0ba4df693a01359592233f641f1f486911ac320a9a
2025-09-07T09:50:26.4667056Z deleted: sha256:662d8c9dfc7db2f5d004293de4f2b7647941dee4c916479ef082d17fcdfd9c47
2025-09-07T09:50:26.4667455Z deleted: sha256:ea5ad443c754124b3a5a209c2663376b4c156947edef1b982a336148bbf9114d
2025-09-07T09:50:26.4667826Z deleted: sha256:284be7504f072e0c04da4e2190e8d0e1de73835ed67be81f3ddd7eafd5d06a3a
2025-09-07T09:50:26.4668224Z deleted: sha256:2f49ff4be65f7ca55de8d7028fb3df7d08232a9f043aa7ba27d9393724286281
2025-09-07T09:50:26.4668618Z deleted: sha256:f63b503fdd1cca198aecefb9eef7ffbeb5fbc723f2a8462f50316e56cd403cbc
2025-09-07T09:50:26.4669025Z deleted: sha256:f9d46e08457013f0e71d608ac3dd95b79c41120060a80baefa684048cc15574e
2025-09-07T09:50:26.4669408Z deleted: sha256:cab76e28615751b6d6a703103b1da790a67cb3a4ee2e8814de51de18ff8b595d
2025-09-07T09:50:26.4670329Z deleted: sha256:0b2d09aa482371591a32563a5db71472822abd096a347967a9bd2a177737109f
2025-09-07T09:50:26.4670719Z deleted: sha256:d306d346d5da05e9fd04284304b1637a0bf01ee97397c688d19d783d5e133de9
2025-09-07T09:50:26.4671090Z deleted: sha256:bb3381a916d410a6e304540bb0796099dc780cd11f5829e734b337e0e79acfe4
2025-09-07T09:50:26.4671455Z deleted: sha256:bcf487c27e826c092985285163fb896e3324460b1774f3eb2a66623cd31e7d87
2025-09-07T09:50:26.4671816Z deleted: sha256:7d13485a9bdc5c0e64ac5085b25f4dded75c60f74090369c1b6f3f546ee37e94
2025-09-07T09:50:26.4672195Z deleted: sha256:55351d98a4197542fa7c78089671f447a6ef88cc554b7fad4fc522e8d4d187b6
2025-09-07T09:50:26.4672567Z deleted: sha256:f884bc0c4f9a994f3b3f1d82205f3a7014b05c84ad0c1c2fa3254d15a44f31e1
2025-09-07T09:50:26.4672942Z deleted: sha256:cdd16785a15239e518604ea9ea31405d5225fa6411d1c6d74d6523bcebf759ab
2025-09-07T09:50:26.4673445Z deleted: sha256:2c5bc1dc49446d7df5784578ae7c99460a93b502aa0c3b9deffbb95ec5216860
2025-09-07T09:50:26.4673819Z deleted: sha256:bae1e956be98416ce7d1a6c2c6ef0917f467238e19291786f8e1fed36fa81956
2025-09-07T09:50:26.4674192Z deleted: sha256:2cb1f002ab1126b0606999a9557b3f7f5da1e453d5376d29d95d60a979a215c4
2025-09-07T09:50:26.4674571Z deleted: sha256:25055a5f67b9bce8fac50ee1508dcb0f862ed154de5ded734e55f60edaca385f
2025-09-07T09:50:26.4674952Z deleted: sha256:98024e2dd34a5899240e41ae14f59c657cdc005040773e6ad7cfe3d67cdac7a8
2025-09-07T09:50:26.4675332Z deleted: sha256:8d2e75659096b4af8a20c3e9a6cce899b6e720f638eacdfd7d41ec8a736efdde
2025-09-07T09:50:26.4675800Z deleted: sha256:7741a6bf043548509c51c32e44734f30dfe07f91ca56c64422b004c3c0444e68
2025-09-07T09:50:26.4676180Z deleted: sha256:e2e63edbd2512e413c388888eabade05a2a7876adf20e7f0e0c3660ac3acbd3d
2025-09-07T09:50:26.4676559Z deleted: sha256:7fdea0f7711ee22084f87dc6d651598b5e5c5237de828105f698cb6a937d4c9c
2025-09-07T09:50:26.4676938Z deleted: sha256:486a2cf42f9492f291d59d48f3cec5a0a72449d8b6ad7d7a02596da237cdd154
2025-09-07T09:50:26.4677318Z deleted: sha256:a17da64c93a4939fad81a3ff6b6cb30f988176a6e0062fcf9c65e06cd9b9c3fb
2025-09-07T09:50:26.4677698Z deleted: sha256:70b4a3a917b8f95b19ae5dab6f404af8fa1c886022e4a1d785654013d5d876af
2025-09-07T09:50:26.4678078Z deleted: sha256:bd1b9d6a8aa636a67023800dcd85e4a3a7a7a21d65c6e6491d169fa65b4404a9
2025-09-07T09:50:26.4678457Z deleted: sha256:e3befcf3d3693c1d7bf0535e6e6722f0aabb0123805443ef5915dd5441ed0b00
2025-09-07T09:50:26.4678837Z deleted: sha256:4b4f846f1c4266b015f5fdf8dac5346c083c3aee2375e337172c112677c5a8c0
2025-09-07T09:50:26.4679208Z deleted: sha256:f05dc4d1350267b90e07af241a64f86a928fb3d8de75717ac04ec5a0433d042f
2025-09-07T09:50:26.4679593Z deleted: sha256:b6b4de696915fa2db09844ec9ac44dbb2940b655cd356404cf1ff03eec644dad
2025-09-07T09:50:26.4679969Z deleted: sha256:da008bbe1fc29cb35b3949040e97eb801f3264a56c4dd1b9d43a3cb54f2a39b2
2025-09-07T09:50:26.4680352Z deleted: sha256:261da5d14cad99ee11dcdaeb6055726f38fc12b7c559ee9c6d2ddc3f288f4828
2025-09-07T09:50:26.4680776Z deleted: sha256:16f900c60e70d685a85ca571ee0dada993a02217bdd6bb8b1d49169e7e28cf41
2025-09-07T09:50:26.4691201Z deleted: sha256:f57b18c5cde1d1dc553a15e1e98141d4afc0b4d0bb1182cc85b2c21bd18bb783
2025-09-07T09:50:26.4691690Z deleted: sha256:3c79105088ac60b231e4553752ee42cb6a87f9d32736b32f0c2123dddec724e7
2025-09-07T09:50:26.4692085Z deleted: sha256:df1ffff478908236efb6ceb8e05e6e078f12b864f4d24ce598cba7b961fad65c
2025-09-07T09:50:26.4692471Z deleted: sha256:8170255b562b59b76768f18a5b84b1ba887db93d3fe43b87a74bdc6be4f82014
2025-09-07T09:50:26.4692851Z deleted: sha256:c863cfe6bed704be5a54617331e27158b6f5a492dd6b9ed9c99d23db017cf5e1
2025-09-07T09:50:26.4693245Z deleted: sha256:e9e5a98c073f72c3abf9cc98724a31a3791535574ac78aeda7eb5df4580b21d0
2025-09-07T09:50:26.4693614Z deleted: sha256:0a42ac98735ca6578911218be7a7918001fe8aee1eb33d98f0d0a153d0e1102d
2025-09-07T09:50:26.4694000Z deleted: sha256:77d5a8aaa4d0fe1210dda9ac1f0fa3cf6141fea925b6240b9839d7505d021d3f
2025-09-07T09:50:26.4694388Z deleted: sha256:fa6ec46c43532dc01449df1cc403de8bb5872f859076e90658534c51c1487ef9
2025-09-07T09:50:26.4694898Z deleted: sha256:424a12dd5083283e19af48d31b7f2e33911ca8f459796f17280eaf5777a9aa25
2025-09-07T09:50:26.4695288Z deleted: sha256:8f0499601e14f1073e20ce889b45d12ab33264f9cf30359ac29dddbf58a311aa
2025-09-07T09:50:26.4695668Z deleted: sha256:5a5fae32dfb81abcd7bf374018b11e8e42a5aa39841d4b94e822d306c9af015b
2025-09-07T09:50:26.4696059Z deleted: sha256:d1bda89f22d383d38dfb7f7590b3bb202ccb91814034e7c7e2493306a10151ef
2025-09-07T09:50:26.4696442Z deleted: sha256:dbf16c1fcae146528685a8f745f9c505b24ba9ef009c42b1bd711ff7bf51b936
2025-09-07T09:50:26.4696819Z deleted: sha256:f9ec0065788f638325536a37427e2635b760a32457f20ca0acbcef6946b1041b
2025-09-07T09:50:26.4697190Z deleted: sha256:9d9911dac8fb2ff7db87329f38625d73f452dfef8822830048bbc00541c7df14
2025-09-07T09:50:26.4697638Z deleted: sha256:de4c1937129850e357b0de484d230569f628ac0bc883b12eff42932cd1e193ce
2025-09-07T09:50:26.4698067Z deleted: sha256:7b3c9e5b56a1d74226a5c1a54e5cb5e749012aa9b1d2376c6e7503757e29c35b
2025-09-07T09:50:26.4698445Z deleted: sha256:8062a6f28fc5fe2a199e1c1c40b6c43b7e29eb0c452492b47ec6900413b19cb6
2025-09-07T09:50:26.4698833Z deleted: sha256:f879aeffe6886f8da80462b571f9307aa63bb961645bec55ff579187a81cfd0b
2025-09-07T09:50:26.4699210Z deleted: sha256:5c6ef06b3536a430194aee509a784ee889c4a9d6248cb20fd9290e87e4ee2245
2025-09-07T09:50:26.4699591Z deleted: sha256:461aea034a25a2d72be6adfe9213c457c4cbf48724e9cb1c57987afb87668f21
2025-09-07T09:50:26.4699979Z deleted: sha256:e342cd1c71b7d0b024ea16b4a11f3f7fbbc2e3d11ef754c9d242aa50c4f8b0a3
2025-09-07T09:50:26.4700372Z deleted: sha256:bffd35a7fa1ddcfe05f79b7d3cae4180928eeea00eaab7ed7f484bc31adfc1d5
2025-09-07T09:50:26.4700803Z deleted: sha256:b34e33e7b04b5cbb5d5852199430593bfa18ddfe9081df42284230a14ebb739e
2025-09-07T09:50:26.4701174Z deleted: sha256:21d9b55338774d9ddc66d0bfcc92af9c8d2ecd94d1710b7049f5a811e411af7b
2025-09-07T09:50:26.4701558Z deleted: sha256:6cc2b33909585d17bf269fb8297ff881249e136137254734f7d23b9583208718
2025-09-07T09:50:26.4701940Z deleted: sha256:ca7f55b7c6d6cb11ddd8e187da34c2695fc2ce7655d652b9c9dc140a01ed056f
2025-09-07T09:50:26.4702334Z deleted: sha256:a3ece3d0ab6e99ef783c4f8d27d0e38504ab4477590ef556c16d22d92ba63a43
2025-09-07T09:50:26.4702712Z deleted: sha256:c137b0d41177c753aa1b69b11d0dd1f82420bf8520371866c845b53dca10b2d0
2025-09-07T09:50:26.4703356Z deleted: sha256:1e0d92b07bce12e511af59f608edd1932b10704d700f5e7538e406b90ecbb615
2025-09-07T09:50:26.4703737Z deleted: sha256:2ec3d01b3031e9da124d67410f54866ec5c679a0d6e4aee6b31608c45ce7fd77
2025-09-07T09:50:26.4704116Z deleted: sha256:308cffbd71363688c672b2043c6b9bf647cfb84593c42c3d88e3f36ee8f7f1b4
2025-09-07T09:50:26.4704496Z deleted: sha256:d965d9873fa450daba50a85d961f0835b14374167d84cfafa6060d16229f4229
2025-09-07T09:50:26.4704875Z deleted: sha256:effd997e222f62a34133bb2ecf9c0ffee151e5797f72e734d86a270d2e722374
2025-09-07T09:50:26.4705272Z deleted: sha256:0bbc1c78c10ee09c2697cfcce347dc9edbf82a7ccc25a6db6ee0a8dda398f7f2
2025-09-07T09:50:26.4705660Z deleted: sha256:214858e773d1ad73c2965c19b29cbfd3e2a974daa879163e1c1eb96567a7ee06
2025-09-07T09:50:26.4706041Z deleted: sha256:a9c7a2cd7ae229b26e84c093de657d0f4334d6cc9301991c6c3245ff62a9a71d
2025-09-07T09:50:26.4706424Z deleted: sha256:749a80551ef3f272e2517cb065bc7a5250da47d0b36bf74ed453caa9a5fee265
2025-09-07T09:50:26.4706797Z deleted: sha256:39b014c4e62d21c11df6c6d775d3f345675014292198981f455bacc4515a0f7b
2025-09-07T09:50:26.4707167Z deleted: sha256:0f087c9a894566644f825f5f87308d92e4cf149c51f7cd4769cbfaeefd3df791
2025-09-07T09:50:26.4707544Z deleted: sha256:dc6eb6dad5f9e332f00af553440e857b1467db1be43dd910cdb6830ba0898d50
2025-09-07T09:50:26.4707768Z 
2025-09-07T09:50:26.4707863Z Total reclaimed space: 62.83GB
2025-09-07T09:50:26.4779885Z Post job cleanup.
2025-09-07T09:50:26.4809917Z Post job cleanup.
2025-09-07T09:50:26.5556194Z [command]/usr/bin/git version
2025-09-07T09:50:26.5598824Z git version 2.47.1
2025-09-07T09:50:26.5629317Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/9fca48f9-8840-46a9-a73f-63f2cb79e3ff/.gitconfig'
2025-09-07T09:50:26.5639041Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9fca48f9-8840-46a9-a73f-63f2cb79e3ff' before making global git config changes
2025-09-07T09:50:26.5639686Z Adding repository directory to the temporary git global config as a safe directory
2025-09-07T09:50:26.5642526Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-09-07T09:50:26.5681081Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-09-07T09:50:26.5723733Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-09-07T09:50:26.6089649Z Entering 'android/libs/fbjni'
2025-09-07T09:50:26.6157569Z Entering 'third_party/FP16'
2025-09-07T09:50:26.6225977Z Entering 'third_party/FXdiv'
2025-09-07T09:50:26.6290153Z Entering 'third_party/NNPACK'
2025-09-07T09:50:26.6354728Z Entering 'third_party/NVTX'
2025-09-07T09:50:26.6422834Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T09:50:26.6496138Z Entering 'third_party/XNNPACK'
2025-09-07T09:50:26.6576124Z Entering 'third_party/aiter'
2025-09-07T09:50:26.6645267Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T09:50:26.6719063Z Entering 'third_party/benchmark'
2025-09-07T09:50:26.6785016Z Entering 'third_party/composable_kernel'
2025-09-07T09:50:26.6857644Z Entering 'third_party/cpp-httplib'
2025-09-07T09:50:26.6923414Z Entering 'third_party/cpuinfo'
2025-09-07T09:50:26.6987764Z Entering 'third_party/cudnn_frontend'
2025-09-07T09:50:26.7054185Z Entering 'third_party/cutlass'
2025-09-07T09:50:26.7133816Z Entering 'third_party/fbgemm'
2025-09-07T09:50:26.7203763Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T09:50:26.7267395Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T09:50:26.7339499Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T09:50:26.7402387Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T09:50:26.7472564Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T09:50:26.7540553Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T09:50:26.7602422Z Entering 'third_party/fbgemm/external/json'
2025-09-07T09:50:26.7670631Z Entering 'third_party/flash-attention'
2025-09-07T09:50:26.7738675Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T09:50:26.7808519Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T09:50:26.7882976Z Entering 'third_party/flatbuffers'
2025-09-07T09:50:26.7952316Z Entering 'third_party/fmt'
2025-09-07T09:50:26.8020558Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T09:50:26.8086006Z Entering 'third_party/gloo'
2025-09-07T09:50:26.8151791Z Entering 'third_party/googletest'
2025-09-07T09:50:26.8217813Z Entering 'third_party/ideep'
2025-09-07T09:50:26.8282257Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T09:50:26.8352466Z Entering 'third_party/ittapi'
2025-09-07T09:50:26.8418364Z Entering 'third_party/kineto'
2025-09-07T09:50:26.8485321Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T09:50:26.8548581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T09:50:26.8613044Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T09:50:26.8675336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T09:50:26.8740742Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T09:50:26.8810789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T09:50:26.8879331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T09:50:26.8943880Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T09:50:26.9013537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T09:50:26.9078192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T09:50:26.9145990Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T09:50:26.9209820Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T09:50:26.9275663Z Entering 'third_party/kleidiai'
2025-09-07T09:50:26.9343583Z Entering 'third_party/mimalloc'
2025-09-07T09:50:26.9408671Z Entering 'third_party/nlohmann'
2025-09-07T09:50:26.9475968Z Entering 'third_party/onnx'
2025-09-07T09:50:26.9557402Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T09:50:26.9626810Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T09:50:26.9693562Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T09:50:26.9759873Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T09:50:26.9824730Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T09:50:26.9888223Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T09:50:26.9955800Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T09:50:27.0019512Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T09:50:27.0082646Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T09:50:27.0145767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T09:50:27.0209941Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T09:50:27.0276385Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T09:50:27.0357555Z Entering 'third_party/pocketfft'
2025-09-07T09:50:27.0424497Z Entering 'third_party/protobuf'
2025-09-07T09:50:27.0494596Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T09:50:27.0560215Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T09:50:27.0626822Z Entering 'third_party/psimd'
2025-09-07T09:50:27.0694704Z Entering 'third_party/pthreadpool'
2025-09-07T09:50:27.0764648Z Entering 'third_party/pybind11'
2025-09-07T09:50:27.0832751Z Entering 'third_party/python-peachpy'
2025-09-07T09:50:27.0900146Z Entering 'third_party/sleef'
2025-09-07T09:50:27.0965947Z Entering 'third_party/tensorpipe'
2025-09-07T09:50:27.1031941Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T09:50:27.1093656Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T09:50:27.1157584Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T09:50:27.1220169Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T09:50:27.1285354Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T09:50:27.1378353Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-09-07T09:50:27.1404166Z http.https://github.com/.extraheader
2025-09-07T09:50:27.1414607Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-09-07T09:50:27.1447178Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-09-07T09:50:27.1799938Z Entering 'android/libs/fbjni'
2025-09-07T09:50:27.1848938Z http.https://github.com/.extraheader
2025-09-07T09:50:27.1889388Z Entering 'third_party/FP16'
2025-09-07T09:50:27.1937463Z http.https://github.com/.extraheader
2025-09-07T09:50:27.1979752Z Entering 'third_party/FXdiv'
2025-09-07T09:50:27.2024507Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2064684Z Entering 'third_party/NNPACK'
2025-09-07T09:50:27.2111017Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2149939Z Entering 'third_party/NVTX'
2025-09-07T09:50:27.2195021Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2238037Z Entering 'third_party/VulkanMemoryAllocator'
2025-09-07T09:50:27.2281841Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2324122Z Entering 'third_party/XNNPACK'
2025-09-07T09:50:27.2370605Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2426768Z Entering 'third_party/aiter'
2025-09-07T09:50:27.2471056Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2513187Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-09-07T09:50:27.2556645Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2605281Z Entering 'third_party/benchmark'
2025-09-07T09:50:27.2648329Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2688278Z Entering 'third_party/composable_kernel'
2025-09-07T09:50:27.2734381Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2780254Z Entering 'third_party/cpp-httplib'
2025-09-07T09:50:27.2824665Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2865597Z Entering 'third_party/cpuinfo'
2025-09-07T09:50:27.2911294Z http.https://github.com/.extraheader
2025-09-07T09:50:27.2951850Z Entering 'third_party/cudnn_frontend'
2025-09-07T09:50:27.2999242Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3042276Z Entering 'third_party/cutlass'
2025-09-07T09:50:27.3086609Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3134941Z Entering 'third_party/fbgemm'
2025-09-07T09:50:27.3180044Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3226872Z Entering 'third_party/fbgemm/external/asmjit'
2025-09-07T09:50:27.3269184Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3310077Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-09-07T09:50:27.3354386Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3399924Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-09-07T09:50:27.3445268Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3486349Z Entering 'third_party/fbgemm/external/cutlass'
2025-09-07T09:50:27.3532453Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3582687Z Entering 'third_party/fbgemm/external/googletest'
2025-09-07T09:50:27.3625498Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3665649Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-09-07T09:50:27.3712071Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3750057Z Entering 'third_party/fbgemm/external/json'
2025-09-07T09:50:27.3794584Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3840838Z Entering 'third_party/flash-attention'
2025-09-07T09:50:27.3886498Z http.https://github.com/.extraheader
2025-09-07T09:50:27.3931001Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-09-07T09:50:27.3973987Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4020642Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-09-07T09:50:27.4063877Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4112052Z Entering 'third_party/flatbuffers'
2025-09-07T09:50:27.4159482Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4201887Z Entering 'third_party/fmt'
2025-09-07T09:50:27.4246526Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4291446Z Entering 'third_party/gemmlowp/gemmlowp'
2025-09-07T09:50:27.4336215Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4377993Z Entering 'third_party/gloo'
2025-09-07T09:50:27.4423780Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4464406Z Entering 'third_party/googletest'
2025-09-07T09:50:27.4510526Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4550207Z Entering 'third_party/ideep'
2025-09-07T09:50:27.4594099Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4635359Z Entering 'third_party/ideep/mkl-dnn'
2025-09-07T09:50:27.4679727Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4727099Z Entering 'third_party/ittapi'
2025-09-07T09:50:27.4771738Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4812223Z Entering 'third_party/kineto'
2025-09-07T09:50:27.4857817Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4898783Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-09-07T09:50:27.4942482Z http.https://github.com/.extraheader
2025-09-07T09:50:27.4983802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-09-07T09:50:27.5027289Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5069735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-09-07T09:50:27.5116247Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5156911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-09-07T09:50:27.5206788Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5246515Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-09-07T09:50:27.5290128Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5330127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-09-07T09:50:27.5371791Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5417501Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-09-07T09:50:27.5461017Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5502880Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-09-07T09:50:27.5547802Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5588258Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-09-07T09:50:27.5635744Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5677626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-09-07T09:50:27.5720915Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5766117Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-09-07T09:50:27.5811436Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5850837Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-09-07T09:50:27.5893061Z http.https://github.com/.extraheader
2025-09-07T09:50:27.5989011Z Entering 'third_party/kleidiai'
2025-09-07T09:50:27.6034065Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6077273Z Entering 'third_party/mimalloc'
2025-09-07T09:50:27.6120453Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6161441Z Entering 'third_party/nlohmann'
2025-09-07T09:50:27.6207659Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6251709Z Entering 'third_party/onnx'
2025-09-07T09:50:27.6296890Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6353414Z Entering 'third_party/onnx/third_party/pybind11'
2025-09-07T09:50:27.6398170Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6446985Z Entering 'third_party/opentelemetry-cpp'
2025-09-07T09:50:27.6491190Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6535101Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-09-07T09:50:27.6576340Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6618431Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-09-07T09:50:27.6660682Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6702274Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-09-07T09:50:27.6746363Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6786456Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-09-07T09:50:27.6831385Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6873176Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-09-07T09:50:27.6917388Z http.https://github.com/.extraheader
2025-09-07T09:50:27.6955544Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-09-07T09:50:27.6998315Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7039324Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-09-07T09:50:27.7081654Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7121638Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-09-07T09:50:27.7162757Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7208166Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-09-07T09:50:27.7252515Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7294945Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-09-07T09:50:27.7340935Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7399206Z Entering 'third_party/pocketfft'
2025-09-07T09:50:27.7444978Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7487217Z Entering 'third_party/protobuf'
2025-09-07T09:50:27.7531776Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7574407Z Entering 'third_party/protobuf/third_party/benchmark'
2025-09-07T09:50:27.7617607Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7661576Z Entering 'third_party/protobuf/third_party/googletest'
2025-09-07T09:50:27.7706283Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7747909Z Entering 'third_party/psimd'
2025-09-07T09:50:27.7790853Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7835367Z Entering 'third_party/pthreadpool'
2025-09-07T09:50:27.7881236Z http.https://github.com/.extraheader
2025-09-07T09:50:27.7920800Z Entering 'third_party/pybind11'
2025-09-07T09:50:27.7965101Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8007669Z Entering 'third_party/python-peachpy'
2025-09-07T09:50:27.8051365Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8093525Z Entering 'third_party/sleef'
2025-09-07T09:50:27.8141244Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8181277Z Entering 'third_party/tensorpipe'
2025-09-07T09:50:27.8228632Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8269175Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-09-07T09:50:27.8311120Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8352012Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-09-07T09:50:27.8396647Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8438032Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-09-07T09:50:27.8482139Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8523971Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-09-07T09:50:27.8567395Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8606224Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-09-07T09:50:27.8648211Z http.https://github.com/.extraheader
2025-09-07T09:50:27.8787967Z A job completed hook has been configured by the self-hosted runner administrator
2025-09-07T09:50:27.8804616Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2025-09-07T09:50:27.8810436Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-09-07T09:50:27.8810682Z ##[endgroup]
2025-09-07T09:50:27.8910307Z [!ALERT!] Swap in detected! [!ALERT!]
2025-09-07T09:50:36.6314317Z [!ALERT!] Swap out detected [!ALERT!]
2025-09-07T09:50:51.8479712Z Cleaning up orphan processes